diff --git a/benchmarks/bridge-fee.yml b/benchmarks/bridge-fee.yml index edbfe71e..12567741 100644 --- a/benchmarks/bridge-fee.yml +++ b/benchmarks/bridge-fee.yml @@ -30,7 +30,7 @@ methodology: findings: [] -source: https://github.com/OpenChainBench/OpenChainBench/tree/main/harnesses/bridge-fee +source: https://github.com/OpenChainBench/OpenChainBench/tree/main/harnesses/bridge-monitor prometheus: url: https://prometheus-production-9ffe.up.railway.app diff --git a/benchmarks/bridge-quote-latency.yml b/benchmarks/bridge-quote-latency.yml index 70c6b163..c136118c 100644 --- a/benchmarks/bridge-quote-latency.yml +++ b/benchmarks/bridge-quote-latency.yml @@ -27,7 +27,7 @@ methodology: findings: [] -source: https://github.com/OpenChainBench/OpenChainBench/tree/main/harnesses/bridge-quote-latency +source: https://github.com/OpenChainBench/OpenChainBench/tree/main/harnesses/bridge-monitor prometheus: url: https://prometheus-production-9ffe.up.railway.app diff --git a/benchmarks/metadata-coverage.yml b/benchmarks/metadata-coverage.yml index 5f3e6294..a24665cd 100644 --- a/benchmarks/metadata-coverage.yml +++ b/benchmarks/metadata-coverage.yml @@ -36,7 +36,7 @@ methodology: findings: [] -source: https://github.com/OpenChainBench/OpenChainBench/tree/main/harnesses/metadata-coverage +source: https://github.com/OpenChainBench/OpenChainBench/tree/main/harnesses/aggregator-head-lag prometheus: url: https://prometheus-production-0859.up.railway.app diff --git a/harnesses/README.md b/harnesses/README.md index be03ea50..45196ceb 100644 --- a/harnesses/README.md +++ b/harnesses/README.md @@ -1,24 +1,30 @@ # Harnesses -One harness folder per benchmark, one Go binary per folder, one Railway service per binary. The Mobula-hosted services emit metrics that the shared OpenChainBench Prometheus scrapes; the site queries that Prometheus. +Each subdirectory is a benchmark runner — a long-running process that calls providers, measures latency / cost / success, and exposes Prometheus metrics on `/metrics`. Anyone can clone, fork, host, or contribute one. ``` harnesses/ -├── aggregator-head-lag/ Bench № 001 (Go, exposes :2112/metrics) -├── bridge-quote-latency/ Bench № 002 (Go, exposes :9090/metrics) -├── bridge-fee/ Bench № 003 (Go, exposes :9090/metrics) -└── metadata-coverage/ Bench № 004 (Go, exposes :2112/metrics) +├── aggregator-head-lag/ Bench № 001 — Go service, exposes :2112/metrics +└── bridge-monitor/ Bench № 002 + № 003 — Go service, exposes :9090/metrics ``` -## Hosting model +A single harness can serve multiple benchmarks when the same set of measurements is consumed by more than one spec — `bridge-monitor` is the canonical example, producing both `bridge_quote_latency_ms` (read by `bridge-quote-latency.yml`) and `bridge_cost_percent` (read by `bridge-fee.yml`). -Federation, not centralization. Each harness is run by whoever owns it. The current four are hosted by Mobula on Railway as a sponsorship contribution; new contributors host their own and submit a scrape config so the central Prometheus picks them up. +## Hosting model — federation, not centralization -You never share API keys with the project. Your harness runs with your credentials, on your infra, on your dime; only the metric values it publishes are public. +OpenChainBench does **not** centralize harness execution. Each harness is hosted by whoever owns it: + +- **Mobula** runs `aggregator-head-lag` and `bridge-monitor` on its own Railway infrastructure (sponsoring those two benchmarks). +- **Independent contributors** run their own harnesses on whatever infra they prefer (Railway, Fly, Cloud Run, a VPS, a home server with a static IP). +- **Providers that want to be benchmarked on their own service** can host the harness themselves and submit a scrape config — the data path is identical to externally-hosted benchmarks. + +The only piece of infrastructure the project shares is a single Prometheus instance ([`/infrastructure/prometheus`](../infrastructure/prometheus)) that scrapes every harness's public `/metrics` endpoint and aggregates everything into one queryable URL. The site queries that URL. + +This means you never share API keys with the project. You run your harness with your own credentials, on your own infra, on your own dime — the maintainers never see your secrets. ## Contract -Every harness, regardless of language, must satisfy this contract: +A harness is a **data producer**, nothing more. It must: | Concern | Requirement | | --- | --- | @@ -26,12 +32,12 @@ Every harness, regardless of language, must satisfy this contract: | Loop | Run continuously and update the same metric set every iteration | | Metric names | Match the names referenced in the matching `benchmarks/.yml` exactly | | Labels | Include `provider` (or equivalent) and `region` at minimum; chain/route labels encouraged | -| Endpoint | Expose `/metrics` over HTTPS on a publicly reachable URL | -| Timeouts | Documented; failures fail closed | +| Endpoint | Expose `/metrics` over HTTPS on a publicly reachable URL. The OpenChainBench Prometheus scrapes from the public internet — no VPN-only addresses | +| Timeouts | Documented; failures fail closed (counted toward success rate, excluded from latency aggregates) | | Reproducibility | README explains how to run locally with one command | | License | MIT, same as the rest of the repo | -A harness does not ship its own Prometheus, Grafana, or Alertmanager. The shared infrastructure handles that. +A harness does **not** ship its own Prometheus, Grafana, Alertmanager, or `docker-compose.yml`. Hosting choice is the contributor's, but the data plane is shared. ## Subdirectory layout @@ -45,4 +51,25 @@ harnesses// ## Submitting a new harness -See [`/CONTRIBUTING.md`](../CONTRIBUTING.md) for the full submission flow. +See [`/CONTRIBUTING.md`](../CONTRIBUTING.md) for the full submission flow. Short version: + +1. Open an issue with the [📊 Propose a benchmark template](https://github.com/OpenChainBench/OpenChainBench/issues/new?template=new-benchmark.yml). +2. Build the harness here at `harnesses//`. +3. Deploy it on whatever infra you prefer, expose `/metrics` over HTTPS at a stable URL. +4. Append a scrape job to [`infrastructure/prometheus/prometheus.yml`](../infrastructure/prometheus/prometheus.yml) pointing at your URL. +5. Write the spec at `benchmarks/.yml`. +6. Open a PR. + +Once merged, a maintainer redeploys the central Prometheus to apply the new scrape job — your benchmark appears on the site within 60 seconds (next ISR cycle). + +## Hosting tips for contributors + +A few options for hosting a harness, ranked roughly from "trivial setup" to "more control": + +- **Railway** (via this repo, root `harnesses//`) — same workflow as how Mobula hosts its harnesses. Free tier covers most light harnesses. +- **Fly.io** — `fly deploy` in any folder with a Dockerfile. Generous free tier. +- **Google Cloud Run** — pay-per-use, scales to zero. +- **VPS (Hetzner, DigitalOcean, OVH)** — €5/mo, full control, run a `systemd` service exposing a port. +- **Your laptop with ngrok** — fine for short-lived experiments, not for live benchmarks. + +Whatever you pick, the OpenChainBench Prometheus only needs `https:///metrics` to be reachable. diff --git a/harnesses/aggregator-head-lag/README.md b/harnesses/aggregator-head-lag/README.md index 78b6c095..1b282f1c 100644 --- a/harnesses/aggregator-head-lag/README.md +++ b/harnesses/aggregator-head-lag/README.md @@ -1,29 +1,31 @@ # Harness · aggregator-head-lag -> Real-time monitor that produces the `head_lag_seconds` metric consumed by [`benchmarks/aggregator-head-lag.yml`](../../benchmarks/aggregator-head-lag.yml). One Go binary, one Railway service, one benchmark. +> Real-time monitor that produces the `head_lag_seconds` metric consumed by [`benchmarks/aggregator-head-lag.yml`](../../benchmarks/aggregator-head-lag.yml). **Bench**: [№ 001 · Aggregator Head Lag](../../benchmarks/aggregator-head-lag.yml) ## How it works -Connects to each aggregator's WebSocket / REST feed and measures latency by comparing: +The monitor connects to each aggregator's WebSocket / REST feed and measures latency by comparing: - The on-chain timestamp of a trade event (from the event payload) - The wall-clock time at which the aggregator emitted the event -The delta is exported as a Prometheus gauge (`head_lag_seconds`), labelled by aggregator, chain and region. Failures surface as `head_lag_errors_total` counters. +The delta is exported as a Prometheus gauge (`head_lag_seconds`), labelled by aggregator, chain and region. Failures surface as `head_lag_errors_total` counters. REST API latency, quote API latency and metadata coverage are recorded in parallel from the same process. **Tracked aggregators**: GeckoTerminal · Mobula · Codex **Supported chains**: Solana · Ethereum · BNB Chain · Base ## Where the data goes -This harness is a data producer only. It exposes `/metrics` on port `2112`. The shared OpenChainBench Prometheus (see [`/infrastructure/prometheus`](../../infrastructure/prometheus)) scrapes that endpoint: +This harness is a **data producer only** — it exposes `/metrics` on port `2112`. The shared OpenChainBench Prometheus (see [`/infrastructure/prometheus`](../../infrastructure/prometheus)) scrapes that endpoint over Railway's internal DNS: ``` aggregator-head-lag.railway.internal:2112 ──► prometheus.railway.internal ──► public site ``` +A contributor running locally just needs `/metrics` reachable from their Prom scraper — nothing else. + ## Metrics produced ``` @@ -35,6 +37,9 @@ mobula_processing_lag_seconds{...} gauge mobula_network_lag_seconds{...} gauge aggregator_head_block{aggregator, chain, region} gauge blockchain_head_block{chain, region} gauge +rest_api_latency_milliseconds_{bucket,sum,count}{...} histogram +quote_api_latency_milliseconds_{bucket,sum,count}{...} histogram +metadata_coverage_total / metadata_coverage_success{...} counter ``` ## Run locally @@ -43,13 +48,13 @@ Prerequisites: Go 1.24+, API keys for the aggregators you want to track. ```bash cp .env.example .env -# Fill in MOBULA_API_KEY, COINGECKO_API_KEY, DEFINED_SESSION_COOKIE +# Fill in the keys (any missing key disables that aggregator's monitor) go run ./cmd/script/ ``` -`/metrics` will be exposed on `http://localhost:2112/metrics`. +`/metrics` will be exposed on `http://localhost:2112/metrics`. To exercise the OpenChainBench site against your local data, point the YAML spec's `prom_url` at a local Prometheus that scrapes this endpoint (the `infrastructure/prometheus` config has notes on how to do this). -Or via Docker: +Or run via Docker: ```bash docker build -t aggregator-head-lag . @@ -58,7 +63,7 @@ docker run --rm --env-file .env -p 2112:2112 aggregator-head-lag ## Run on Railway -This service is deployed from the OpenChainBench repo, root directory `harnesses/aggregator-head-lag/`. Set the env vars listed below and the shared Prometheus picks it up via DNS. +This service is deployed from the OpenChainBench repo, root directory `harnesses/aggregator-head-lag/`. Set the env vars listed below, and the shared Prometheus will pick it up via DNS automatically. ## Environment variables @@ -70,21 +75,48 @@ This service is deployed from the OpenChainBench repo, root directory `harnesses | `MOBULA_WS_URL` | Override the Mobula WS endpoint | optional | | `MONITOR_REGION` | Label written on every metric (e.g. `us-east`) | recommended | -A monitor with no key for a given aggregator is skipped cleanly. +A monitor with no key for a given aggregator is skipped cleanly — the harness will run for whatever providers it can authenticate with. ## Project layout ``` -cmd/script/ Single Go binary - ├── main.go Boots the head-lag goroutine + metrics server - ├── config.go Env-var loader - ├── metrics.go Prometheus metric definitions - ├── head_lag_monitor.go WebSocket monitor: on-chain ↔ feed timestamp delta - ├── mobula_fast_trade_monitor.go Mobula-specific lag breakdown - ├── proxy.go HTTP transport + retries - └── log_buffer.go In-memory ring buffer +cmd/script/ Single Go binary — all monitors + Prometheus exporter + ├── main.go Entrypoint, supervises goroutines per aggregator + ├── config.go Env-var loader + ├── metrics.go Prometheus metric definitions + HTTP /metrics handler + ├── head_lag_monitor.go WebSocket monitor: on-chain ↔ feed timestamp delta + ├── mobula_*.go Mobula REST + WS monitors + ├── codex_*.go Codex REST monitor + ├── geckoterminal_monitor.go + ├── quote_api_monitor.go Mobula swap quoting latency + ├── metadata_coverage_monitor.go + ├── proxy.go HTTP transport + retries + └── log_buffer.go In-memory ring buffer (last N log lines) + +Dockerfile Multi-stage Go build +.env.example Documented env vars ``` +## Adding an aggregator + +1. Create `cmd/script/_monitor.go` mirroring an existing file (e.g. `geckoterminal_monitor.go`). +2. Implement the WebSocket / REST loop and call the appropriate `metrics.go` recorder. +3. Add the API key field to `Config` in `config.go` and to `.env.example`. +4. Start the monitor goroutine in `main.go`. +5. Update the YAML spec at `benchmarks/aggregator-head-lag.yml` to reference the new aggregator under `providers:` and add its query under `prometheus.providers.`. + +## Troubleshooting + +```bash +# Are metrics being produced? +curl http://localhost:2112/metrics | grep head_lag_seconds + +# Tail logs (Docker) +docker logs -f +``` + +If a specific aggregator silently emits nothing: most often the API key is missing or rate-limited. Check the logs for `[AGGREGATOR][skip]` lines. + ## License -MIT, same as the rest of OpenChainBench. +MIT — same as the rest of OpenChainBench. diff --git a/harnesses/aggregator-head-lag/cmd/script/main.go b/harnesses/aggregator-head-lag/cmd/script/main.go index 84444379..c0a7eaa7 100644 --- a/harnesses/aggregator-head-lag/cmd/script/main.go +++ b/harnesses/aggregator-head-lag/cmd/script/main.go @@ -47,17 +47,50 @@ func main() { } }() - // Head lag monitor — the only data producer for this binary. - // Measures the gap between an on-chain Swap event and its appearance - // on each aggregator's WebSocket / REST feed. + // Mobula Pulse V2 feeder — only feeds the metadata coverage queue, + // no pulse-specific metrics emitted (see mobula_pulse_monitor.go). + wg.Add(1) + go func() { + defer wg.Done() + runMobulaPulseMonitor(config, stopChan) + }() + + // Mobula REST API monitor + wg.Add(1) + go func() { + defer wg.Done() + runMobulaRESTMonitor(config, stopChan) + }() + + // Codex REST API monitor + wg.Add(1) + go func() { + defer wg.Done() + runCodexRESTMonitor(config, stopChan) + }() + + // Quote API latency monitor (Jupiter, Li.Fi, 1inch, KyberSwap) + wg.Add(1) + go func() { + defer wg.Done() + runQuoteAPIMonitor(config, stopChan) + }() + + // Metadata coverage monitor (Mobula vs Codex) + wg.Add(1) + go func() { + defer wg.Done() + runMetadataCoverageMonitor(config, stopChan) + }() + + // Head lag monitor (blockchain head vs aggregator indexed head) wg.Add(1) go func() { defer wg.Done() runHeadLagMonitor(config, stopChan) }() - // Mobula Fast-Trade monitor — used for the detailed Mobula breakdown - // (processing vs network lag) emitted alongside head lag. + // Mobula Fast-Trade monitor (for comparison with Pulse V2) wg.Add(1) go func() { defer wg.Done() diff --git a/harnesses/bridge-fee/README.md b/harnesses/bridge-fee/README.md deleted file mode 100644 index e792bb36..00000000 --- a/harnesses/bridge-fee/README.md +++ /dev/null @@ -1,48 +0,0 @@ -# Harness · bridge-fee - -> Bridge cost-percent monitor that produces `bridge_cost_percent` metrics consumed by [`benchmarks/bridge-fee.yml`](../../benchmarks/bridge-fee.yml). One Go binary, one Railway service, one benchmark. - -**Bench**: [№ 003 · Bridge Effective Fee](../../benchmarks/bridge-fee.yml) - -**Tracked bridges**: Mobula · Relay · Li.Fi · Debridge - -## How it works - -Quote loop runs every 5 minutes: for every route × every notional, each bridge is queried and the total cost (fees + slippage + destination gas) is computed and exported as a percent of notional. - -We use `bridge_cost_percent` rather than the narrower `bridge_fees_percent` because some providers bake their fee into the spread and would otherwise appear at 0% — misleading. `bridge_cost_percent` reflects what actually leaves the user's wallet. - -## Metrics produced (subset relevant to this bench) - -``` -bridge_cost_percent{bridge, from_chain, to_chain, from_token, to_token, amount_usd, region} gauge -bridge_cost_usd{...} gauge -bridge_slippage_usd / bridge_gas_usd / bridge_fix_fee_usd{...} gauge -``` - -## Run locally - -```bash -cp .env.example .env -docker-compose -f deploy/docker-compose.yml up -d -``` - -Or directly: - -```bash -go run ./cmd/monitor/ -# /metrics on :9090 -``` - -## Environment variables - -| Var | Required | Notes | -| --- | --- | --- | -| `MOBULA_API_KEY` | yes | Quotes | -| `LIFI_API_KEY` | recommended | Better rate limits | -| `EXECUTION_MODE` | yes | `dry-run` (recommended for this service) | -| `MONITOR_REGION` | yes | Metric label | - -## License - -MIT. diff --git a/harnesses/bridge-fee/.gitignore b/harnesses/bridge-monitor/.gitignore similarity index 100% rename from harnesses/bridge-fee/.gitignore rename to harnesses/bridge-monitor/.gitignore diff --git a/harnesses/bridge-fee/Dockerfile b/harnesses/bridge-monitor/Dockerfile similarity index 100% rename from harnesses/bridge-fee/Dockerfile rename to harnesses/bridge-monitor/Dockerfile diff --git a/harnesses/bridge-monitor/README.md b/harnesses/bridge-monitor/README.md new file mode 100644 index 00000000..872031c9 --- /dev/null +++ b/harnesses/bridge-monitor/README.md @@ -0,0 +1,223 @@ +# Harness · bridge-monitor + +> Real-world benchmark of cross-chain bridges. Produces every metric consumed by both [`benchmarks/bridge-quote-latency.yml`](../../benchmarks/bridge-quote-latency.yml) and [`benchmarks/bridge-fee.yml`](../../benchmarks/bridge-fee.yml) — quote latency, cost percent, fees, success rate, end-to-end execution latency. + +**Benches**: + +- [№ 002 · Bridge — Quote Latency](../../benchmarks/bridge-quote-latency.yml) +- [№ 003 · Bridge — Cost Percent](../../benchmarks/bridge-fee.yml) + +**Tracked bridges**: Mobula · Relay · Li.Fi (executed) · Debridge (quote-only — flat fees too high for small tickets) + +## How it works + +Two loops run simultaneously inside one Go binary: + +1. **Quote loop** (free, every 5 minutes) — fetches a quote from every bridge for every route × every notional ($5 / $50 / $300). Records quote latency, fees, slippage, estimated time. All four bridges benchmarked here. +2. **Execution loop** (paid, fixed UTC times) — actually broadcasts real transactions through Mobula + Relay + Li.Fi. Measures quote → broadcast → settlement end-to-end latency and revert rates. Debridge is excluded from execution. + +Plus daily P&L cron, wallet balance metrics, pre-flight tier simulation, and per-tier Slack notifications. + +## Where the data goes + +This harness is a **data producer only** — it exposes `/metrics` on port `9090` (overridable via `METRICS_PORT`). The shared OpenChainBench Prometheus (see [`/infrastructure/prometheus`](../../infrastructure/prometheus)) scrapes that endpoint: + +``` +bridge-monitor.railway.internal:9090 ──► prometheus.railway.internal ──► public site +``` + +## Test routes + +### USDC triangle + +``` + Solana USDC + ↗ ↘ + R3 R1 + ↗ ↘ +Arbitrum USDT ←──R2─── Base USDC +``` + +Capital circulates through the triangle — only bridge fees + gas are consumed. Each tier executes one bridge at a time round-trip, then the next bridge starts a fresh triangle with the replenished wallet (peak capital = 1× tier per leg, not 3×). + +| Route | From | To | Schedule | +| --- | --- | --- | --- | +| R1 | USDC (Solana) | USDC (Base) | Tiers $5/$50/$300 | +| R2 | USDC (Base) | USDT (Arbitrum) | Tiers $5/$50/$300 | +| R3 | USDT (Arbitrum) | USDC (Solana) | Tiers $5/$50/$300 | +| R4 | TRUMP (Solana) | BRETT (Base) | Weekly $5 only (one-way) | +| R5 | USDC (Arbitrum) | USDC (HyperCore) | Quote-only $5/$50/$300 | + +## Metrics produced + +### Bridge performance + +| Metric | Description | +| --- | --- | +| `bridge_quote_latency_ms` | Time to receive quote | +| `bridge_execution_latency_ms` | Broadcast → funds received | +| `bridge_e2e_latency_ms` | Quote start → funds received | +| `bridge_fees_usd` / `bridge_fees_percent` | Bridge fees only | +| `bridge_cost_usd` / `bridge_cost_percent` | Total cost = fees + slippage + destination gas | +| `bridge_slippage_usd` / `bridge_gas_usd` / `bridge_fix_fee_usd` | Cost breakdown | +| `bridge_estimated_time_ms` | Bridge's own promise from the quote | +| `bridge_output_usd` | USD received on destination | +| `bridge_quote_success` | 1 on success, 0 on error | +| `bridge_success_total` / `bridge_reverts_total` / `bridge_errors_total` | Outcome counters | +| `bridge_consecutive_failures` | Streak length (resets on success) | + +### Wallet state + +| Metric | Description | +| --- | --- | +| `wallet_balance_usd{chain,token}` | Current USD per leg, refreshed every 5 min | +| `wallet_balance_last_update_timestamp_seconds` | Last refresh timestamp | + +The site reads these via the YAML specs — `bridge-quote-latency.yml` reads `bridge_quote_latency_ms_*`, `bridge-fee.yml` reads `bridge_cost_percent`. + +## Run locally + +```bash +cp .env.example .env +# Fill in API keys, wallet keys (or leave dry-run defaults) +go run ./cmd/monitor/ +``` + +`/metrics` will be exposed on `http://localhost:9090/metrics`. + +If `EXECUTION_MODE=dry-run` (default) the monitor only fetches quotes — no transactions, no fees, no wallets needed. Quote-loop metrics still flow normally. + +Or via Docker: + +```bash +docker build -t bridge-monitor . +docker run --rm --env-file .env -p 9090:9090 bridge-monitor +``` + +## Execution modes + +| Mode | What it does | +| --- | --- | +| `dry-run` | Quotes only. Default. No TX, no fees. | +| `single-test` | One real TX per route at `TEST_AMOUNT_USD` (default $1) then exits. | +| `production` | Full loop + scheduler. Real TX at 10:00 UTC. | + +```bash +go run ./cmd/monitor/ # dry-run +EXECUTION_MODE=single-test TEST_AMOUNT_USD=0.50 go run ./cmd/monitor/ +EXECUTION_MODE=production go run ./cmd/monitor/ +``` + +## Run on Railway + +This service is deployed from the OpenChainBench repo, root directory `harnesses/bridge-monitor/`. Set the env vars listed below; the shared Prometheus will pick it up via DNS automatically. + +## Project layout + +``` +cmd/monitor/ Main binary — quote loop, execution, scheduler + ├── main.go + ├── config.go Env loading + ├── metrics.go Prometheus metric definitions + HTTP /metrics handler + ├── scheduler.go Fixed UTC time scheduling + ├── executor.go Per-route balance check, exec orchestration + ├── cycle_sim.go Pre-flight viability simulation + ├── tx_executor.go TX signing + broadcast (overridable RPCs) + ├── wallet.go Wallet manager + ├── balance.go Balance fetch + Prometheus export + ├── onchain_balance.go Post-fill balance reader (EVM + Solana + HL) + ├── wallet_snapshot.go Daily 09:30 P&L cron + tier health grid + ├── slack.go Per-exec, skip, P&L, startup notifs + ├── pricer.go USD pricing + ├── mobula_bridge.go Mobula integration + ├── relay_bridge.go Relay integration + ├── lifi_bridge.go Li.Fi integration + └── debridge_bridge.go Debridge integration (quote-only) + +cmd/rebalance/ Manual one-shot rebalance tool (unused in prod) + +Dockerfile Multi-stage Go build +.env.example Documented env vars +``` + +## Bridge integration details + +### Mobula + +- Quote: `GET /api/2/bridge/quote?...&apiKey=...` +- TX data: `deposit.solana.serializedTx` (Solana) or `deposit.evm` (EVM) +- 2-step EVM (approve + deposit) handled +- Status: `GET /api/2/bridge/status/{txHash}`, terminal `"settled"` + +### Relay + +- Quote: `POST /quote` JSON body +- TX data: `steps[].items[].data.instructions[]` (Solana) or `steps[].items[].data` (EVM) +- 2-step EVM (approve + deposit) handled — iterates `steps[]` to find approve before deposit +- Request ID: `steps[].requestId` (deposit step) +- Status: `GET /intents/status/v3?requestId=...` + +### Li.Fi + +- Quote: `GET /v1/quote?...` with `x-lifi-api-key` header +- TX data: `transactionRequest.data` (Solana base64) or `transactionRequest` (EVM) +- Approval address: `estimate.approvalAddress` (EVM) +- Status: `GET /v1/status?txHash=...&fromChain=...&toChain=...` + +### Debridge + +- Quote only — execution disabled because flat native-token fees make $5 tickets unprofitable. + +## Wallet funding (production mode only) + +| Chain | Token | Minimum | Purpose | +| --- | --- | --- | --- | +| Solana | SOL | $5 | Gas (~$0.01/TX) | +| Solana | USDC | $T for tier $T | R1 source | +| Solana | TRUMP | $20+ | R4 (weekly $5) | +| Base | USDC | $0+ (R1 auto-fills before R2 needs it) | R2 source | +| Base | ETH | $5 | Gas (~$0.10/TX) | +| Arbitrum | USDT | $0+ (R2 auto-fills before R3 needs it) | R3 source | +| Arbitrum | ETH | $5 | Gas (~$0.10/TX) | + +Pre-flight `SimulateTriangleCycle` checks the 1× threshold before every scheduled tier — non-viable tiers skip cleanly with a single Slack message. + +## Environment variables + +| Var | Required | Notes | +| --- | --- | --- | +| `MOBULA_API_KEY` | yes | Quotes + balance API | +| `LIFI_API_KEY` | recommended | Better rate limits | +| `RELAY_API_KEY` | optional | Public quotes work without it | +| `WALLET_EVM_PRIVATE_KEY` | for prod | Hex `0x...` | +| `WALLET_EVM_ADDRESS` | for prod | | +| `WALLET_SOL_PRIVATE_KEY` | for prod | Base58 | +| `WALLET_SOL_ADDRESS` | for prod | | +| `EXECUTION_MODE` | yes | `dry-run` / `single-test` / `production` | +| `TEST_AMOUNT_USD` | for `single-test` | Default $1 | +| `MAX_DAILY_SPEND_USD` | no | Safety cap, default $10 | +| `MONITOR_REGION` | yes | Metric label | +| `METRICS_PORT` | no | Default 9090 | +| `SLACK_WEBHOOK_URL` | optional | Direct Slack notifs from the monitor | +| `SOLANA_RPC` / `BASE_RPC` / `ARB_RPC` | no | Override public RPCs | + +## Safety features + +1. **Per-tier pre-flight** — full cycle simulated before execution, skip cleanly if not viable. +2. **Per-route balance check** — secondary guard during `RunReal` (1.05× buffer). +3. **Daily spending cap** — `MAX_DAILY_SPEND_USD` hard stops runaway. +4. **Consecutive-failure streak** — 3 fails in a row on the same bridge → critical alert. +5. **Dry-run / single-test default** — no accidental production runs. +6. **EVM revert detection** — TX receipt inspected. +7. **Gas price +50 % bump** on Arbitrum — handles rapid base-fee variance. + +## Adding a bridge + +1. Create `cmd/monitor/_bridge.go` mirroring an existing integration (e.g. `relay_bridge.go`). +2. Implement quote, execute, status methods conforming to the `Bridge` interface. +3. Add API key field to `Config` and `.env.example`. +4. Register the bridge in `main.go`'s bridge list. + +## License + +MIT — same as the rest of OpenChainBench. diff --git a/harnesses/bridge-fee/cmd/rebalance/main.go b/harnesses/bridge-monitor/cmd/rebalance/main.go similarity index 100% rename from harnesses/bridge-fee/cmd/rebalance/main.go rename to harnesses/bridge-monitor/cmd/rebalance/main.go diff --git a/harnesses/bridge-fee/cmd/rebalance/mobula.go b/harnesses/bridge-monitor/cmd/rebalance/mobula.go similarity index 100% rename from harnesses/bridge-fee/cmd/rebalance/mobula.go rename to harnesses/bridge-monitor/cmd/rebalance/mobula.go diff --git a/harnesses/bridge-fee/cmd/rebalance/relay_simple.go b/harnesses/bridge-monitor/cmd/rebalance/relay_simple.go similarity index 100% rename from harnesses/bridge-fee/cmd/rebalance/relay_simple.go rename to harnesses/bridge-monitor/cmd/rebalance/relay_simple.go diff --git a/harnesses/bridge-fee/cmd/rebalance/tx_executor.go b/harnesses/bridge-monitor/cmd/rebalance/tx_executor.go similarity index 100% rename from harnesses/bridge-fee/cmd/rebalance/tx_executor.go rename to harnesses/bridge-monitor/cmd/rebalance/tx_executor.go diff --git a/harnesses/bridge-fee/go.mod b/harnesses/bridge-monitor/go.mod similarity index 100% rename from harnesses/bridge-fee/go.mod rename to harnesses/bridge-monitor/go.mod diff --git a/harnesses/bridge-fee/go.sum b/harnesses/bridge-monitor/go.sum similarity index 100% rename from harnesses/bridge-fee/go.sum rename to harnesses/bridge-monitor/go.sum diff --git a/harnesses/bridge-quote-latency/.gitignore b/harnesses/bridge-quote-latency/.gitignore deleted file mode 100644 index 13b984a0..00000000 --- a/harnesses/bridge-quote-latency/.gitignore +++ /dev/null @@ -1,7 +0,0 @@ -.env -bin/ -*.log -prometheus_data/ -grafana_data/ -.DS_Store -monitor diff --git a/harnesses/bridge-quote-latency/Dockerfile b/harnesses/bridge-quote-latency/Dockerfile deleted file mode 100644 index 75a62cad..00000000 --- a/harnesses/bridge-quote-latency/Dockerfile +++ /dev/null @@ -1,26 +0,0 @@ -FROM golang:1.24-alpine AS builder - -WORKDIR /build - -# Copy go mod files -COPY go.mod go.sum ./ -RUN go mod download - -# Copy source code -COPY . . - -# Build the binary -RUN CGO_ENABLED=0 GOOS=linux go build -o /bridge-monitor ./cmd/monitor - -# Final stage -FROM alpine:latest - -RUN apk --no-cache add ca-certificates - -WORKDIR /app - -COPY --from=builder /bridge-monitor . - -EXPOSE 9090 - -CMD ["./bridge-monitor"] diff --git a/harnesses/bridge-quote-latency/README.md b/harnesses/bridge-quote-latency/README.md deleted file mode 100644 index 732e24b2..00000000 --- a/harnesses/bridge-quote-latency/README.md +++ /dev/null @@ -1,48 +0,0 @@ -# Harness · bridge-quote-latency - -> Bridge quote-latency monitor that produces `bridge_quote_latency_ms_*` metrics consumed by [`benchmarks/bridge-quote-latency.yml`](../../benchmarks/bridge-quote-latency.yml). One Go binary, one Railway service, one benchmark. - -**Bench**: [№ 002 · Bridge Quote Latency](../../benchmarks/bridge-quote-latency.yml) - -**Tracked bridges**: Mobula · Relay · Li.Fi · Debridge - -## How it works - -Quote loop runs every 5 minutes: for every route × every notional ($5 / $50 / $300), each bridge is queried and the wall-clock time to receive a usable quote is recorded as a histogram observation. - -The same quote loop emits cost metrics consumed by the sibling `bridge-fee` harness, but that benchmark is served by its own service so the two stay independent. - -## Metrics produced (subset relevant to this bench) - -``` -bridge_quote_latency_ms_{bucket,sum,count}{bridge, from_chain, to_chain, from_token, to_token, amount_usd, region} histogram -bridge_quote_success{bridge, ...} gauge 0/1 -bridge_errors_total{bridge, error_type, ...} counter -``` - -## Run locally - -```bash -cp .env.example .env -docker-compose -f deploy/docker-compose.yml up -d -``` - -Or directly: - -```bash -go run ./cmd/monitor/ -# /metrics on :9090 -``` - -## Environment variables - -| Var | Required | Notes | -| --- | --- | --- | -| `MOBULA_API_KEY` | yes | Quotes | -| `LIFI_API_KEY` | recommended | Better rate limits | -| `EXECUTION_MODE` | yes | `dry-run` (recommended for this service) | -| `MONITOR_REGION` | yes | Metric label | - -## License - -MIT. diff --git a/harnesses/bridge-quote-latency/cmd/rebalance/main.go b/harnesses/bridge-quote-latency/cmd/rebalance/main.go deleted file mode 100644 index de5f8be2..00000000 --- a/harnesses/bridge-quote-latency/cmd/rebalance/main.go +++ /dev/null @@ -1,193 +0,0 @@ -package main - -import ( - "bufio" - "fmt" - "log" - "os" - "strings" - "time" -) - -// Test 2 — manual Mobula HyperCore deposit benchmark. -// Bridges $5 of Base USDC into the EVM wallet's Hyperliquid perp account. -// Used once to validate the on-chain fill measurement path before wiring the -// HL clearinghouse balance reader into the main monitor. - -func main() { - log.Println("🧪 Test 2 — Mobula Base USDC → HyperCore deposit") - log.Println("━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━") - - config := loadEnvSimple() - if config.mobulaAPIKey == "" || config.evmPrivateKey == "" { - log.Fatal("❌ Missing required env vars (MOBULA_API_KEY, WALLET_EVM_PRIVATE_KEY)") - } - - amountUSD := 5.0 - if v := os.Getenv("TEST_AMOUNT_USD"); v != "" { - fmt.Sscanf(v, "%f", &amountUSD) - } - - log.Printf("EVM Address (sender + HC recipient): %s", config.evmAddress) - log.Printf("Plan: Base USDC $%.2f → HyperCore (hl:mainnet) USDC via Mobula", amountUSD) - log.Println("\n⚠️ Real funds, real TX. Costs ~$0.10 gas, $5 capital moves to HL.") - log.Print("Continue? (yes/no): ") - - reader := bufio.NewReader(os.Stdin) - input, _ := reader.ReadString('\n') - if strings.TrimSpace(input) != "yes" { - log.Println("Aborted.") - return - } - - mobula := NewMobulaBridgeSimple(config.mobulaAPIKey) - txExec, err := NewTxExecutorSimple(config.solPrivateKey, config.evmPrivateKey, config.mobulaAPIKey) - if err != nil { - log.Fatalf("❌ Failed to init TX executor: %v", err) - } - - if err := executeBridge(mobula, txExec, BridgeParams{ - FromChain: "evm:8453", // Base - FromToken: "0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913", // Base USDC - ToChain: "hl:mainnet", - ToToken: "USDC", - Amount: amountUSD, - SenderAddr: config.evmAddress, - ReceiverAddr: config.evmAddress, - }); err != nil { - log.Fatalf("❌ Test 2 failed: %v", err) - } - - log.Println("\n⏳ Waiting 15s for HL indexer to credit the perp account...") - time.Sleep(15 * time.Second) - log.Println("\n✅ Test 2 deposit complete. Run the post-snapshot curl now to read withdrawable on HL.") -} - -type simpleConfig struct { - mobulaAPIKey string - evmAddress string - solAddress string - evmPrivateKey string - solPrivateKey string -} - -func loadEnvSimple() *simpleConfig { - config := &simpleConfig{} - config.mobulaAPIKey = os.Getenv("MOBULA_API_KEY") - config.evmAddress = os.Getenv("WALLET_EVM_ADDRESS") - config.solAddress = os.Getenv("WALLET_SOL_ADDRESS") - config.evmPrivateKey = os.Getenv("WALLET_EVM_PRIVATE_KEY") - config.solPrivateKey = os.Getenv("WALLET_SOL_PRIVATE_KEY") - - file, err := os.Open(".env") - if err == nil { - defer file.Close() - scanner := bufio.NewScanner(file) - for scanner.Scan() { - line := strings.TrimSpace(scanner.Text()) - if line == "" || strings.HasPrefix(line, "#") { - continue - } - parts := strings.SplitN(line, "=", 2) - if len(parts) != 2 { - continue - } - key, value := strings.TrimSpace(parts[0]), strings.TrimSpace(parts[1]) - value = strings.Trim(value, `"'`) - switch key { - case "MOBULA_API_KEY": - if config.mobulaAPIKey == "" { - config.mobulaAPIKey = value - } - case "WALLET_EVM_ADDRESS": - if config.evmAddress == "" { - config.evmAddress = value - } - case "WALLET_SOL_ADDRESS": - if config.solAddress == "" { - config.solAddress = value - } - case "WALLET_EVM_PRIVATE_KEY": - if config.evmPrivateKey == "" { - config.evmPrivateKey = value - } - case "WALLET_SOL_PRIVATE_KEY": - if config.solPrivateKey == "" { - config.solPrivateKey = value - } - } - } - } - return config -} - -type BridgeParams struct { - FromChain string - FromToken string - ToChain string - ToToken string - Amount float64 - SenderAddr string - ReceiverAddr string - IsSolanaSource bool -} - -func executeBridge(mobula *MobulaBridgeSimple, txExec *TxExecutorSimple, params BridgeParams) error { - log.Printf("\n📝 Getting quote for $%.2f...", params.Amount) - quote, err := mobula.GetQuote(params) - if err != nil { - return fmt.Errorf("quote failed: %w", err) - } - log.Printf("✅ Quote received: fee=$%s, output=$%s, time=%dms", - quote.Data.Fees.TotalFeeUsd, quote.Data.EstimatedAmountOutUsd, 0) - - approveIdx := -1 - for i, step := range quote.Data.Steps { - if step.Type == "approve" { - approveIdx = i - break - } - } - - if approveIdx >= 0 { - step := quote.Data.Steps[approveIdx] - log.Printf("📝 Sending approval TX → %s", step.Tx.To) - approvalHash, err := txExec.ExecuteEVMTx(params.FromChain, step.Tx.To, step.Tx.Data, step.Tx.Value) - if err != nil { - return fmt.Errorf("approval failed: %w", err) - } - log.Printf("✅ Approval TX: %s", approvalHash) - log.Printf("⏳ Waiting 15s for confirmation...") - time.Sleep(15 * time.Second) - } - - var txHash string - if quote.Data.Deposit.EVM.To != "" { - log.Printf("📤 Broadcasting EVM bridge TX → %s", quote.Data.Deposit.EVM.To) - txHash, err = txExec.ExecuteEVMTx(params.FromChain, - quote.Data.Deposit.EVM.To, quote.Data.Deposit.EVM.Data, quote.Data.Deposit.EVM.Value) - } else { - for _, step := range quote.Data.Steps { - if step.Type != "approve" { - log.Printf("📤 Broadcasting EVM bridge TX (from steps) → %s", step.Tx.To) - txHash, err = txExec.ExecuteEVMTx(params.FromChain, step.Tx.To, step.Tx.Data, step.Tx.Value) - break - } - } - } - if err != nil { - return fmt.Errorf("broadcast failed: %w", err) - } - log.Printf("✅ Bridge TX broadcast: %s", txHash) - - log.Printf("⏳ Polling Mobula status (timeout 5min)...") - status, err := txExec.PollMobulaStatus(txHash, 5*time.Minute) - if err != nil { - return fmt.Errorf("status poll failed: %w", err) - } - if status.Data.Status == "filled" || status.Data.Status == "settled" { - log.Printf("🎉 Bridge complete! Status: %s | TX: https://basescan.org/tx/%s", status.Data.Status, txHash) - return nil - } - return fmt.Errorf("unexpected final status: %s", status.Data.Status) -} diff --git a/harnesses/bridge-quote-latency/cmd/rebalance/mobula.go b/harnesses/bridge-quote-latency/cmd/rebalance/mobula.go deleted file mode 100644 index 61f7f7dc..00000000 --- a/harnesses/bridge-quote-latency/cmd/rebalance/mobula.go +++ /dev/null @@ -1,105 +0,0 @@ -package main - -import ( - "encoding/json" - "fmt" - "io" - "net/http" - "strconv" - "time" -) - -type MobulaBridgeSimple struct { - apiKey string - client *http.Client -} - -type MobulaQuoteResp struct { - Data struct { - EstimatedAmountOut string `json:"estimatedAmountOut"` - EstimatedAmountOutUsd string `json:"estimatedAmountOutUsd"` - Fees struct { - TotalFeeUsd string `json:"totalFeeUsd"` - } `json:"fees"` - Deposit struct { - Solana struct { - SerializedTx string `json:"serializedTx"` - } `json:"solana"` - EVM struct { - To string `json:"to"` - Data string `json:"data"` - Value string `json:"value"` - } `json:"evm"` - } `json:"deposit"` - Steps []struct { - Type string `json:"type"` - Tx struct { - To string `json:"to"` - Data string `json:"data"` - Value string `json:"value"` - } `json:"tx"` - } `json:"steps"` - } `json:"data"` -} - -type MobulaStatusResp struct { - Data struct { - Status string `json:"status"` - } `json:"data"` -} - -func NewMobulaBridgeSimple(apiKey string) *MobulaBridgeSimple { - return &MobulaBridgeSimple{ - apiKey: apiKey, - client: &http.Client{Timeout: 45 * time.Second}, - } -} - -func (m *MobulaBridgeSimple) GetQuote(params BridgeParams) (*MobulaQuoteResp, error) { - url := fmt.Sprintf( - "https://api.mobula.io/api/2/bridge/quote?originChainId=%s&originToken=%s&destinationChainId=%s&destinationToken=%s&amount=%s&walletAddress=%s&senderAddress=%s&apiKey=%s", - params.FromChain, params.FromToken, - params.ToChain, params.ToToken, - strconv.FormatFloat(params.Amount, 'f', -1, 64), - params.ReceiverAddr, params.SenderAddr, - m.apiKey, - ) - - resp, err := m.client.Get(url) - if err != nil { - return nil, err - } - defer resp.Body.Close() - - body, _ := io.ReadAll(resp.Body) - if resp.StatusCode != http.StatusOK { - return nil, fmt.Errorf("API error %d: %s", resp.StatusCode, string(body)) - } - - var quote MobulaQuoteResp - if err := json.Unmarshal(body, "e); err != nil { - return nil, err - } - - return "e, nil -} - -func (m *MobulaBridgeSimple) GetStatus(txHash string) (*MobulaStatusResp, error) { - url := fmt.Sprintf("https://api.mobula.io/api/2/bridge/status/%s", txHash) - - req, _ := http.NewRequest("GET", url, nil) - req.Header.Set("Authorization", m.apiKey) - - resp, err := m.client.Do(req) - if err != nil { - return nil, err - } - defer resp.Body.Close() - - var status MobulaStatusResp - if err := json.NewDecoder(resp.Body).Decode(&status); err != nil { - return nil, err - } - - return &status, nil -} diff --git a/harnesses/bridge-quote-latency/cmd/rebalance/relay_simple.go b/harnesses/bridge-quote-latency/cmd/rebalance/relay_simple.go deleted file mode 100644 index 653f0a92..00000000 --- a/harnesses/bridge-quote-latency/cmd/rebalance/relay_simple.go +++ /dev/null @@ -1,94 +0,0 @@ -package main - -import ( - "bytes" - "encoding/json" - "fmt" - "io" - "net/http" - "strings" - "time" -) - -type RelayQuoteReq struct { - User string `json:"user"` - OriginChainID int64 `json:"originChainId"` - DestinationChainID int64 `json:"destinationChainId"` - OriginCurrency string `json:"originCurrency"` - DestinationCurrency string `json:"destinationCurrency"` - Amount string `json:"amount"` - TradeType string `json:"tradeType"` - Recipient string `json:"recipient"` -} - -type RelayQuoteResp struct { - Details struct { - CurrencyIn struct{ AmountUsd string `json:"amountUsd"` } `json:"currencyIn"` - CurrencyOut struct{ AmountUsd string `json:"amountUsd"` } `json:"currencyOut"` - } `json:"details"` - Steps []struct { - ID string `json:"id"` - RequestId string `json:"requestId"` - Items []struct { - Data struct { - To string `json:"to"` - Data string `json:"data"` - Value string `json:"value"` - } `json:"data"` - Check struct { - Endpoint string `json:"endpoint"` - } `json:"check"` - } `json:"items"` - } `json:"steps"` -} - -type RelayStatus struct { - Status string `json:"status"` - Details string `json:"details"` -} - -func relayQuote(req RelayQuoteReq) (*RelayQuoteResp, error) { - body, _ := json.Marshal(req) - httpReq, _ := http.NewRequest("POST", "https://api.relay.link/quote", bytes.NewReader(body)) - httpReq.Header.Set("Content-Type", "application/json") - client := &http.Client{Timeout: 45 * time.Second} - resp, err := client.Do(httpReq) - if err != nil { - return nil, err - } - defer resp.Body.Close() - raw, _ := io.ReadAll(resp.Body) - if resp.StatusCode != http.StatusOK { - return nil, fmt.Errorf("relay %d: %s", resp.StatusCode, string(raw)) - } - var out RelayQuoteResp - if err := json.Unmarshal(raw, &out); err != nil { - return nil, err - } - return &out, nil -} - -func relayPollStatus(checkURL string, timeout time.Duration) (*RelayStatus, error) { - deadline := time.Now().Add(timeout) - client := &http.Client{Timeout: 10 * time.Second} - for time.Now().Before(deadline) { - resp, err := client.Get(checkURL) - if err != nil { - time.Sleep(5 * time.Second) - continue - } - raw, _ := io.ReadAll(resp.Body) - resp.Body.Close() - var s RelayStatus - if err := json.Unmarshal(raw, &s); err == nil { - fmt.Printf(" Status: %s\n", s.Status) - if strings.EqualFold(s.Status, "success") || strings.EqualFold(s.Status, "filled") || - strings.EqualFold(s.Status, "settled") || strings.EqualFold(s.Status, "refund") || - strings.EqualFold(s.Status, "failure") { - return &s, nil - } - } - time.Sleep(5 * time.Second) - } - return nil, fmt.Errorf("timeout polling Relay status") -} diff --git a/harnesses/bridge-quote-latency/cmd/rebalance/tx_executor.go b/harnesses/bridge-quote-latency/cmd/rebalance/tx_executor.go deleted file mode 100644 index 8d7091a6..00000000 --- a/harnesses/bridge-quote-latency/cmd/rebalance/tx_executor.go +++ /dev/null @@ -1,197 +0,0 @@ -package main - -import ( - "context" - "encoding/base64" - "encoding/hex" - "fmt" - "log" - "math/big" - "strings" - "time" - - "github.com/ethereum/go-ethereum/common" - "github.com/ethereum/go-ethereum/core/types" - "github.com/ethereum/go-ethereum/crypto" - "github.com/ethereum/go-ethereum/ethclient" - "github.com/gagliardetto/solana-go" - "github.com/gagliardetto/solana-go/rpc" -) - -type TxExecutorSimple struct { - solanaKey solana.PrivateKey - evmKey []byte - mobulaAPIKey string - mobula *MobulaBridgeSimple -} - -var chainRPCs = map[string]string{ - "evm:8453": "https://mainnet.base.org", - "evm:42161": "https://arb1.arbitrum.io/rpc", -} - -func NewTxExecutorSimple(solPrivKey, evmPrivKey, mobulaAPIKey string) (*TxExecutorSimple, error) { - // Parse Solana key - solKey, err := solana.PrivateKeyFromBase58(solPrivKey) - if err != nil { - return nil, fmt.Errorf("invalid solana key: %w", err) - } - - // Parse EVM key - evmKeyHex := strings.TrimPrefix(evmPrivKey, "0x") - evmKey, err := hex.DecodeString(evmKeyHex) - if err != nil { - return nil, fmt.Errorf("invalid evm key: %w", err) - } - - return &TxExecutorSimple{ - solanaKey: solKey, - evmKey: evmKey, - mobulaAPIKey: mobulaAPIKey, - mobula: NewMobulaBridgeSimple(mobulaAPIKey), - }, nil -} - -func (tx *TxExecutorSimple) ExecuteSolanaTx(serializedTxBase64 string) (string, error) { - // Decode base64 TX - txBytes, err := base64.StdEncoding.DecodeString(serializedTxBase64) - if err != nil { - return "", fmt.Errorf("failed to decode tx: %w", err) - } - - // Deserialize transaction - transaction, err := solana.TransactionFromBytes(txBytes) - if err != nil { - return "", fmt.Errorf("failed to deserialize tx: %w", err) - } - - // Connect to Solana - client := rpc.New("https://api.mainnet-beta.solana.com") - - // Get fresh blockhash (the one in the TX may have expired) - recent, err := client.GetLatestBlockhash(context.Background(), rpc.CommitmentFinalized) - if err != nil { - return "", fmt.Errorf("failed to get blockhash: %w", err) - } - transaction.Message.RecentBlockhash = recent.Value.Blockhash - - // Clear old signatures and re-sign - transaction.Signatures = nil - _, err = transaction.Sign(func(key solana.PublicKey) *solana.PrivateKey { - if key.Equals(tx.solanaKey.PublicKey()) { - return &tx.solanaKey - } - return nil - }) - if err != nil { - return "", fmt.Errorf("failed to sign tx: %w", err) - } - - // Send transaction - sig, err := client.SendTransaction(context.Background(), transaction) - if err != nil { - return "", fmt.Errorf("failed to send tx: %w", err) - } - - return sig.String(), nil -} - -func (tx *TxExecutorSimple) ExecuteEVMTx(chain, to, data, value string) (string, error) { - rpcURL, ok := chainRPCs[chain] - if !ok { - return "", fmt.Errorf("unknown chain: %s", chain) - } - - client, err := ethclient.Dial(rpcURL) - if err != nil { - return "", fmt.Errorf("failed to connect: %w", err) - } - defer client.Close() - - // Parse private key - privateKey, err := crypto.ToECDSA(tx.evmKey) - if err != nil { - return "", fmt.Errorf("invalid key: %w", err) - } - - fromAddr := crypto.PubkeyToAddress(privateKey.PublicKey) - toAddr := common.HexToAddress(to) - - // Parse data - dataBytes, err := hex.DecodeString(strings.TrimPrefix(data, "0x")) - if err != nil { - return "", fmt.Errorf("invalid data: %w", err) - } - - // Parse value - valueBig := new(big.Int) - if value != "" && value != "0" { - value = strings.TrimPrefix(value, "0x") - valueBig.SetString(value, 16) - } - - // Get nonce - nonce, err := client.PendingNonceAt(context.Background(), fromAddr) - if err != nil { - return "", fmt.Errorf("failed to get nonce: %w", err) - } - - // Get gas price + 50% buffer to handle base-fee variance (esp. Arbitrum) - gasPrice, err := client.SuggestGasPrice(context.Background()) - if err != nil { - return "", fmt.Errorf("failed to get gas price: %w", err) - } - gasPrice = new(big.Int).Mul(gasPrice, big.NewInt(150)) - gasPrice = new(big.Int).Div(gasPrice, big.NewInt(100)) - - // Estimate gas - gasLimit := uint64(500000) // Safe default - - // Get chain ID - chainID, err := client.NetworkID(context.Background()) - if err != nil { - return "", fmt.Errorf("failed to get chain ID: %w", err) - } - - // Create transaction - txData := types.NewTransaction(nonce, toAddr, valueBig, gasLimit, gasPrice, dataBytes) - - // Sign transaction - signedTx, err := types.SignTx(txData, types.NewEIP155Signer(chainID), privateKey) - if err != nil { - return "", fmt.Errorf("failed to sign: %w", err) - } - - // Send transaction - err = client.SendTransaction(context.Background(), signedTx) - if err != nil { - return "", fmt.Errorf("failed to send: %w", err) - } - - return signedTx.Hash().Hex(), nil -} - -func (tx *TxExecutorSimple) PollMobulaStatus(txHash string, timeout time.Duration) (*MobulaStatusResp, error) { - deadline := time.Now().Add(timeout) - pollInterval := 5 * time.Second - - for time.Now().Before(deadline) { - status, err := tx.mobula.GetStatus(txHash) - if err != nil { - log.Printf(" Status poll error: %v", err) - time.Sleep(pollInterval) - continue - } - - log.Printf(" Status: %s", status.Data.Status) - - if status.Data.Status == "filled" || status.Data.Status == "settled" || - status.Data.Status == "refunded" || status.Data.Status == "failed" { - return status, nil - } - - time.Sleep(pollInterval) - } - - return nil, fmt.Errorf("timeout waiting for status") -} diff --git a/harnesses/bridge-quote-latency/go.mod b/harnesses/bridge-quote-latency/go.mod deleted file mode 100644 index 043718fd..00000000 --- a/harnesses/bridge-quote-latency/go.mod +++ /dev/null @@ -1,71 +0,0 @@ -module bridging-latency - -go 1.24.0 - -toolchain go1.24.4 - -require ( - github.com/ethereum/go-ethereum v1.17.2 - github.com/gagliardetto/solana-go v1.18.0 - github.com/prometheus/client_golang v1.20.5 -) - -require ( - filippo.io/edwards25519 v1.2.0 // indirect - github.com/Microsoft/go-winio v0.6.2 // indirect - github.com/ProjectZKM/Ziren/crates/go-runtime/zkvm_runtime v0.0.0-20251001021608-1fe7b43fc4d6 // indirect - github.com/StackExchange/wmi v1.2.1 // indirect - github.com/benbjohnson/clock v1.3.5 // indirect - github.com/beorn7/perks v1.0.1 // indirect - github.com/bits-and-blooms/bitset v1.20.0 // indirect - github.com/blendle/zapdriver v1.3.1 // indirect - github.com/cespare/xxhash/v2 v2.3.0 // indirect - github.com/consensys/gnark-crypto v0.18.1 // indirect - github.com/crate-crypto/go-eth-kzg v1.5.0 // indirect - github.com/davecgh/go-spew v1.1.1 // indirect - github.com/deckarep/golang-set/v2 v2.6.0 // indirect - github.com/decred/dcrd/dcrec/secp256k1/v4 v4.0.1 // indirect - github.com/ethereum/c-kzg-4844/v2 v2.1.6 // indirect - github.com/fatih/color v1.18.0 // indirect - github.com/gagliardetto/binary v0.8.0 // indirect - github.com/gagliardetto/treeout v0.1.4 // indirect - github.com/go-logr/logr v1.4.3 // indirect - github.com/go-logr/stdr v1.2.2 // indirect - github.com/go-ole/go-ole v1.3.0 // indirect - github.com/google/uuid v1.6.0 // indirect - github.com/gorilla/websocket v1.5.3 // indirect - github.com/holiman/uint256 v1.3.2 // indirect - github.com/json-iterator/go v1.1.12 // indirect - github.com/klauspost/compress v1.18.0 // indirect - github.com/logrusorgru/aurora v2.0.3+incompatible // indirect - github.com/mattn/go-colorable v0.1.14 // indirect - github.com/mattn/go-isatty v0.0.20 // indirect - github.com/mitchellh/go-testing-interface v1.14.1 // indirect - github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect - github.com/modern-go/reflect2 v1.0.2 // indirect - github.com/mostynb/zstdpool-freelist v0.0.0-20201229113212-927304c0c3b1 // indirect - github.com/mr-tron/base58 v1.2.0 // indirect - github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect - github.com/prometheus/client_model v0.6.1 // indirect - github.com/prometheus/common v0.55.0 // indirect - github.com/prometheus/procfs v0.15.1 // indirect - github.com/shirou/gopsutil v3.21.4-0.20210419000835-c7a38de76ee5+incompatible // indirect - github.com/streamingfast/logging v0.0.0-20250404134358-92b15d2fbd2e // indirect - github.com/supranational/blst v0.3.16 // indirect - github.com/tklauser/go-sysconf v0.3.12 // indirect - github.com/tklauser/numcpus v0.6.1 // indirect - go.mongodb.org/mongo-driver/v2 v2.5.0 // indirect - go.opentelemetry.io/auto/sdk v1.2.1 // indirect - go.opentelemetry.io/otel v1.40.0 // indirect - go.opentelemetry.io/otel/metric v1.40.0 // indirect - go.opentelemetry.io/otel/trace v1.40.0 // indirect - go.uber.org/multierr v1.11.0 // indirect - go.uber.org/ratelimit v0.3.1 // indirect - go.uber.org/zap v1.27.0 // indirect - golang.org/x/crypto v0.47.0 // indirect - golang.org/x/sync v0.18.0 // indirect - golang.org/x/sys v0.40.0 // indirect - golang.org/x/term v0.39.0 // indirect - golang.org/x/time v0.11.0 // indirect - google.golang.org/protobuf v1.36.11 // indirect -) diff --git a/harnesses/bridge-quote-latency/go.sum b/harnesses/bridge-quote-latency/go.sum deleted file mode 100644 index b4e068d8..00000000 --- a/harnesses/bridge-quote-latency/go.sum +++ /dev/null @@ -1,309 +0,0 @@ -filippo.io/edwards25519 v1.2.0 h1:crnVqOiS4jqYleHd9vaKZ+HKtHfllngJIiOpNpoJsjo= -filippo.io/edwards25519 v1.2.0/go.mod h1:xzAOLCNug/yB62zG1bQ8uziwrIqIuxhctzJT18Q77mc= -github.com/AlekSi/pointer v1.2.0 h1:glcy/gc4h8HnG2Z3ZECSzZ1IX1x2JxRVuDzaJwQE0+w= -github.com/AlekSi/pointer v1.2.0/go.mod h1:gZGfd3dpW4vEc/UlyfKKi1roIqcCgwOIvb0tSNSBle0= -github.com/DataDog/zstd v1.4.5 h1:EndNeuB0l9syBZhut0wns3gV1hL8zX8LIu6ZiVHWLIQ= -github.com/DataDog/zstd v1.4.5/go.mod h1:1jcaCB/ufaK+sKp1NBhlGmpz41jOoPQ35bpF36t7BBo= -github.com/Microsoft/go-winio v0.6.2 h1:F2VQgta7ecxGYO8k3ZZz3RS8fVIXVxONVUPlNERoyfY= -github.com/Microsoft/go-winio v0.6.2/go.mod h1:yd8OoFMLzJbo9gZq8j5qaps8bJ9aShtEA8Ipt1oGCvU= -github.com/ProjectZKM/Ziren/crates/go-runtime/zkvm_runtime v0.0.0-20251001021608-1fe7b43fc4d6 h1:1zYrtlhrZ6/b6SAjLSfKzWtdgqK0U+HtH/VcBWh1BaU= -github.com/ProjectZKM/Ziren/crates/go-runtime/zkvm_runtime v0.0.0-20251001021608-1fe7b43fc4d6/go.mod h1:ioLG6R+5bUSO1oeGSDxOV3FADARuMoytZCSX6MEMQkI= -github.com/StackExchange/wmi v1.2.1 h1:VIkavFPXSjcnS+O8yTq7NI32k0R5Aj+v39y29VYDOSA= -github.com/StackExchange/wmi v1.2.1/go.mod h1:rcmrprowKIVzvc+NUiLncP2uuArMWLCbu9SBzvHz7e8= -github.com/VictoriaMetrics/fastcache v1.13.0 h1:AW4mheMR5Vd9FkAPUv+NH6Nhw+fmbTMGMsNAoA/+4G0= -github.com/VictoriaMetrics/fastcache v1.13.0/go.mod h1:hHXhl4DA2fTL2HTZDJFXWgW0LNjo6B+4aj2Wmng3TjU= -github.com/benbjohnson/clock v1.1.0/go.mod h1:J11/hYXuz8f4ySSvYwY0FKfm+ezbsZBKZxNJlLklBHA= -github.com/benbjohnson/clock v1.3.5 h1:VvXlSJBzZpA/zum6Sj74hxwYI2DIxRWuNIoXAzHZz5o= -github.com/benbjohnson/clock v1.3.5/go.mod h1:J11/hYXuz8f4ySSvYwY0FKfm+ezbsZBKZxNJlLklBHA= -github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM= -github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw= -github.com/bits-and-blooms/bitset v1.20.0 h1:2F+rfL86jE2d/bmw7OhqUg2Sj/1rURkBn3MdfoPyRVU= -github.com/bits-and-blooms/bitset v1.20.0/go.mod h1:7hO7Gc7Pp1vODcmWvKMRA9BNmbv6a/7QIWpPxHddWR8= -github.com/blendle/zapdriver v1.3.1 h1:C3dydBOWYRiOk+B8X9IVZ5IOe+7cl+tGOexN4QqHfpE= -github.com/blendle/zapdriver v1.3.1/go.mod h1:mdXfREi6u5MArG4j9fewC+FGnXaBR+T4Ox4J2u4eHCc= -github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs= -github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs= -github.com/cockroachdb/errors v1.11.3 h1:5bA+k2Y6r+oz/6Z/RFlNeVCesGARKuC6YymtcDrbC/I= -github.com/cockroachdb/errors v1.11.3/go.mod h1:m4UIW4CDjx+R5cybPsNrRbreomiFqt8o1h1wUVazSd8= -github.com/cockroachdb/fifo v0.0.0-20240606204812-0bbfbd93a7ce h1:giXvy4KSc/6g/esnpM7Geqxka4WSqI1SZc7sMJFd3y4= -github.com/cockroachdb/fifo v0.0.0-20240606204812-0bbfbd93a7ce/go.mod h1:9/y3cnZ5GKakj/H4y9r9GTjCvAFta7KLgSHPJJYc52M= -github.com/cockroachdb/logtags v0.0.0-20230118201751-21c54148d20b h1:r6VH0faHjZeQy818SGhaone5OnYfxFR/+AzdY3sf5aE= -github.com/cockroachdb/logtags v0.0.0-20230118201751-21c54148d20b/go.mod h1:Vz9DsVWQQhf3vs21MhPMZpMGSht7O/2vFW2xusFUVOs= -github.com/cockroachdb/pebble v1.1.5 h1:5AAWCBWbat0uE0blr8qzufZP5tBjkRyy/jWe1QWLnvw= -github.com/cockroachdb/pebble v1.1.5/go.mod h1:17wO9el1YEigxkP/YtV8NtCivQDgoCyBg5c4VR/eOWo= -github.com/cockroachdb/redact v1.1.5 h1:u1PMllDkdFfPWaNGMyLD1+so+aq3uUItthCFqzwPJ30= -github.com/cockroachdb/redact v1.1.5/go.mod h1:BVNblN9mBWFyMyqK1k3AAiSxhvhfK2oOZZ2lK+dpvRg= -github.com/cockroachdb/tokenbucket v0.0.0-20230807174530-cc333fc44b06 h1:zuQyyAKVxetITBuuhv3BI9cMrmStnpT18zmgmTxunpo= -github.com/cockroachdb/tokenbucket v0.0.0-20230807174530-cc333fc44b06/go.mod h1:7nc4anLGjupUW/PeY5qiNYsdNXj7zopG+eqsS7To5IQ= -github.com/consensys/gnark-crypto v0.18.1 h1:RyLV6UhPRoYYzaFnPQA4qK3DyuDgkTgskDdoGqFt3fI= -github.com/consensys/gnark-crypto v0.18.1/go.mod h1:L3mXGFTe1ZN+RSJ+CLjUt9x7PNdx8ubaYfDROyp2Z8c= -github.com/cpuguy83/go-md2man/v2 v2.0.5 h1:ZtcqGrnekaHpVLArFSe4HK5DoKx1T0rq2DwVB0alcyc= -github.com/cpuguy83/go-md2man/v2 v2.0.5/go.mod h1:tgQtvFlXSQOSOSIRvRPT7W67SCa46tRHOmNcaadrF8o= -github.com/crate-crypto/go-eth-kzg v1.5.0 h1:FYRiJMJG2iv+2Dy3fi14SVGjcPteZ5HAAUe4YWlJygc= -github.com/crate-crypto/go-eth-kzg v1.5.0/go.mod h1:J9/u5sWfznSObptgfa92Jq8rTswn6ahQWEuiLHOjCUI= -github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= -github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= -github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= -github.com/dchest/siphash v1.2.3 h1:QXwFc8cFOR2dSa/gE6o/HokBMWtLUaNDVd+22aKHeEA= -github.com/dchest/siphash v1.2.3/go.mod h1:0NvQU092bT0ipiFN++/rXm69QG9tVxLAlQHIXMPAkHc= -github.com/deckarep/golang-set/v2 v2.6.0 h1:XfcQbWM1LlMB8BsJ8N9vW5ehnnPVIw0je80NsVHagjM= -github.com/deckarep/golang-set/v2 v2.6.0/go.mod h1:VAky9rY/yGXJOLEDv3OMci+7wtDpOF4IN+y82NBOac4= -github.com/decred/dcrd/crypto/blake256 v1.0.0 h1:/8DMNYp9SGi5f0w7uCm6d6M4OU2rGFK09Y2A4Xv7EE0= -github.com/decred/dcrd/crypto/blake256 v1.0.0/go.mod h1:sQl2p6Y26YV+ZOcSTP6thNdn47hh8kt6rqSlvmrXFAc= -github.com/decred/dcrd/dcrec/secp256k1/v4 v4.0.1 h1:YLtO71vCjJRCBcrPMtQ9nqBsqpA1m5sE92cU+pd5Mcc= -github.com/decred/dcrd/dcrec/secp256k1/v4 v4.0.1/go.mod h1:hyedUtir6IdtD/7lIxGeCxkaw7y45JueMRL4DIyJDKs= -github.com/emicklei/dot v1.6.2 h1:08GN+DD79cy/tzN6uLCT84+2Wk9u+wvqP+Hkx/dIR8A= -github.com/emicklei/dot v1.6.2/go.mod h1:DeV7GvQtIw4h2u73RKBkkFdvVAz0D9fzeJrgPW6gy/s= -github.com/ethereum/c-kzg-4844/v2 v2.1.6 h1:xQymkKCT5E2Jiaoqf3v4wsNgjZLY0lRSkZn27fRjSls= -github.com/ethereum/c-kzg-4844/v2 v2.1.6/go.mod h1:8HMkUZ5JRv4hpw/XUrYWSQNAUzhHMg2UDb/U+5m+XNw= -github.com/ethereum/go-bigmodexpfix v0.0.0-20250911101455-f9e208c548ab h1:rvv6MJhy07IMfEKuARQ9TKojGqLVNxQajaXEp/BoqSk= -github.com/ethereum/go-bigmodexpfix v0.0.0-20250911101455-f9e208c548ab/go.mod h1:IuLm4IsPipXKF7CW5Lzf68PIbZ5yl7FFd74l/E0o9A8= -github.com/ethereum/go-ethereum v1.17.2 h1:ag6geu0kn8Hv5FLKTpH+Hm2DHD+iuFtuqKxEuwUsDOI= -github.com/ethereum/go-ethereum v1.17.2/go.mod h1:KHcRXfGOUfUmKg51IhQ0IowiqZ6PqZf08CMtk0g5K1o= -github.com/fatih/color v1.18.0 h1:S8gINlzdQ840/4pfAwic/ZE0djQEH3wM94VfqLTZcOM= -github.com/fatih/color v1.18.0/go.mod h1:4FelSpRwEGDpQ12mAdzqdOukCy4u8WUtOY6lkT/6HfU= -github.com/ferranbt/fastssz v0.1.4 h1:OCDB+dYDEQDvAgtAGnTSidK1Pe2tW3nFV40XyMkTeDY= -github.com/ferranbt/fastssz v0.1.4/go.mod h1:Ea3+oeoRGGLGm5shYAeDgu6PGUlcvQhE2fILyD9+tGg= -github.com/gagliardetto/binary v0.8.0 h1:U9ahc45v9HW0d15LoN++vIXSJyqR/pWw8DDlhd7zvxg= -github.com/gagliardetto/binary v0.8.0/go.mod h1:2tfj51g5o9dnvsc+fL3Jxr22MuWzYXwx9wEoN0XQ7/c= -github.com/gagliardetto/solana-go v1.18.0 h1:+D9KfL5DiuBdhsc1DhlnUU2jww0nSlIPjGBDWNCBxRU= -github.com/gagliardetto/solana-go v1.18.0/go.mod h1:lIYMhelbrx7OaiysurLQS9hC8bqpzAH4OZ51nZo2uaA= -github.com/gagliardetto/treeout v0.1.4 h1:ozeYerrLCmCubo1TcIjFiOWTTGteOOHND1twdFpgwaw= -github.com/gagliardetto/treeout v0.1.4/go.mod h1:loUefvXTrlRG5rYmJmExNryyBRh8f89VZhmMOyCyqok= -github.com/getsentry/sentry-go v0.27.0 h1:Pv98CIbtB3LkMWmXi4Joa5OOcwbmnX88sF5qbK3r3Ps= -github.com/getsentry/sentry-go v0.27.0/go.mod h1:lc76E2QywIyW8WuBnwl8Lc4bkmQH4+w1gwTf25trprY= -github.com/go-logr/logr v1.2.2/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A= -github.com/go-logr/logr v1.4.3 h1:CjnDlHq8ikf6E492q6eKboGOC0T8CDaOvkHCIg8idEI= -github.com/go-logr/logr v1.4.3/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY= -github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag= -github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre4VKE= -github.com/go-ole/go-ole v1.2.5/go.mod h1:pprOEPIfldk/42T2oK7lQ4v4JSDwmV0As9GaiUsvbm0= -github.com/go-ole/go-ole v1.3.0 h1:Dt6ye7+vXGIKZ7Xtk4s6/xVdGDQynvom7xCFEdWr6uE= -github.com/go-ole/go-ole v1.3.0/go.mod h1:5LS6F96DhAwUc7C+1HLexzMXY1xGRSryjyPPKW6zv78= -github.com/gofrs/flock v0.12.1 h1:MTLVXXHf8ekldpJk3AKicLij9MdwOWkZ+a/jHHZby9E= -github.com/gofrs/flock v0.12.1/go.mod h1:9zxTsyu5xtJ9DK+1tFZyibEV7y3uwDxPPfbxeeHCoD0= -github.com/gogo/protobuf v1.3.2 h1:Ov1cvc58UF3b5XjBnZv7+opcTcQFZebYjWzi34vdm4Q= -github.com/gogo/protobuf v1.3.2/go.mod h1:P1XiOD3dCwIKUDQYPy72D8LYyHL2YPYrpS2s69NZV8Q= -github.com/golang-jwt/jwt/v4 v4.5.2 h1:YtQM7lnr8iZ+j5q71MGKkNw9Mn7AjHM68uc9g5fXeUI= -github.com/golang-jwt/jwt/v4 v4.5.2/go.mod h1:m21LjoU+eqJr34lmDMbreY2eSTRJ1cv77w39/MY0Ch0= -github.com/golang/snappy v1.0.0 h1:Oy607GVXHs7RtbggtPBnr2RmDArIsAefDwvrdWvRhGs= -github.com/golang/snappy v1.0.0/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q= -github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8= -github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU= -github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg= -github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0= -github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= -github.com/gorilla/websocket v1.5.3 h1:saDtZ6Pbx/0u+bgYQ3q96pZgCzfhKXGPqt7kZ72aNNg= -github.com/gorilla/websocket v1.5.3/go.mod h1:YR8l580nyteQvAITg2hZ9XVh4b55+EU/adAjf1fMHhE= -github.com/grafana/pyroscope-go v1.2.7 h1:VWBBlqxjyR0Cwk2W6UrE8CdcdD80GOFNutj0Kb1T8ac= -github.com/grafana/pyroscope-go v1.2.7/go.mod h1:o/bpSLiJYYP6HQtvcoVKiE9s5RiNgjYTj1DhiddP2Pc= -github.com/grafana/pyroscope-go/godeltaprof v0.1.9 h1:c1Us8i6eSmkW+Ez05d3co8kasnuOY813tbMN8i/a3Og= -github.com/grafana/pyroscope-go/godeltaprof v0.1.9/go.mod h1:2+l7K7twW49Ct4wFluZD3tZ6e0SjanjcUUBPVD/UuGU= -github.com/hashicorp/go-bexpr v0.1.10 h1:9kuI5PFotCboP3dkDYFr/wi0gg0QVbSNz5oFRpxn4uE= -github.com/hashicorp/go-bexpr v0.1.10/go.mod h1:oxlubA2vC/gFVfX1A6JGp7ls7uCDlfJn732ehYYg+g0= -github.com/holiman/billy v0.0.0-20250707135307-f2f9b9aae7db h1:IZUYC/xb3giYwBLMnr8d0TGTzPKFGNTCGgGLoyeX330= -github.com/holiman/billy v0.0.0-20250707135307-f2f9b9aae7db/go.mod h1:xTEYN9KCHxuYHs+NmrmzFcnvHMzLLNiGFafCb1n3Mfg= -github.com/holiman/bloomfilter/v2 v2.0.3 h1:73e0e/V0tCydx14a0SCYS/EWCxgwLZ18CZcZKVu0fao= -github.com/holiman/bloomfilter/v2 v2.0.3/go.mod h1:zpoh+gs7qcpqrHr3dB55AMiJwo0iURXE7ZOP9L9hSkA= -github.com/holiman/uint256 v1.3.2 h1:a9EgMPSC1AAaj1SZL5zIQD3WbwTuHrMGOerLjGmM/TA= -github.com/holiman/uint256 v1.3.2/go.mod h1:EOMSn4q6Nyt9P6efbI3bueV4e1b3dGlUCXeiRV4ng7E= -github.com/huin/goupnp v1.3.0 h1:UvLUlWDNpoUdYzb2TCn+MuTWtcjXKSza2n6CBdQ0xXc= -github.com/huin/goupnp v1.3.0/go.mod h1:gnGPsThkYa7bFi/KWmEysQRf48l2dvR5bxr2OFckNX8= -github.com/jackpal/go-nat-pmp v1.0.2 h1:KzKSgb7qkJvOUTqYl9/Hg/me3pWgBmERKrTGD7BdWus= -github.com/jackpal/go-nat-pmp v1.0.2/go.mod h1:QPH045xvCAeXUZOxsnwmrtiCoxIr9eob+4orBN1SBKc= -github.com/json-iterator/go v1.1.12 h1:PV8peI4a0ysnczrg+LtxykD8LfKY9ML6u2jnxaEnrnM= -github.com/json-iterator/go v1.1.12/go.mod h1:e30LSqwooZae/UwlEbR2852Gd8hjQvJoHmT4TnhNGBo= -github.com/klauspost/compress v1.11.4/go.mod h1:aoV0uJVorq1K+umq18yTdKaF57EivdYsUV+/s2qKfXs= -github.com/klauspost/compress v1.18.0 h1:c/Cqfb0r+Yi+JtIEq73FWXVkRonBlf0CRNYc8Zttxdo= -github.com/klauspost/compress v1.18.0/go.mod h1:2Pp+KzxcywXVXMr50+X0Q/Lsb43OQHYWRCY2AiWywWQ= -github.com/klauspost/cpuid/v2 v2.0.9 h1:lgaqFMSdTdQYdZ04uHyN2d/eKdOMyi2YLSvlQIBFYa4= -github.com/klauspost/cpuid/v2 v2.0.9/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg= -github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo= -github.com/kr/pretty v0.2.1/go.mod h1:ipq/a2n7PKx3OHsz4KJII5eveXtPO4qwEXGdVfWzfnI= -github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE= -github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk= -github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ= -github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI= -github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY= -github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE= -github.com/kylelemons/godebug v1.1.0 h1:RPNrshWIDI6G2gRW9EHilWtl7Z6Sb1BR0xunSBf0SNc= -github.com/kylelemons/godebug v1.1.0/go.mod h1:9/0rRGxNHcop5bhtWyNeEfOS8JIWk580+fNqagV/RAw= -github.com/leanovate/gopter v0.2.11 h1:vRjThO1EKPb/1NsDXuDrzldR28RLkBflWYcU9CvzWu4= -github.com/leanovate/gopter v0.2.11/go.mod h1:aK3tzZP/C+p1m3SPRE4SYZFGP7jjkuSI4f7Xvpt0S9c= -github.com/logrusorgru/aurora v2.0.3+incompatible h1:tOpm7WcpBTn4fjmVfgpQq0EfczGlG91VSDkswnjF5A8= -github.com/logrusorgru/aurora v2.0.3+incompatible/go.mod h1:7rIyQOR62GCctdiQpZ/zOJlFyk6y+94wXzv6RNZgaR4= -github.com/mattn/go-colorable v0.1.14 h1:9A9LHSqF/7dyVVX6g0U9cwm9pG3kP9gSzcuIPHPsaIE= -github.com/mattn/go-colorable v0.1.14/go.mod h1:6LmQG8QLFO4G5z1gPvYEzlUgJ2wF+stgPZH1UqBm1s8= -github.com/mattn/go-isatty v0.0.20 h1:xfD0iDuEKnDkl03q4limB+vH+GxLEtL/jb4xVJSWWEY= -github.com/mattn/go-isatty v0.0.20/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D7dTCTo3Y= -github.com/minio/sha256-simd v1.0.0 h1:v1ta+49hkWZyvaKwrQB8elexRqm6Y0aMLjCNsrYxo6g= -github.com/minio/sha256-simd v1.0.0/go.mod h1:OuYzVNI5vcoYIAmbIvHPl3N3jUzVedXbKy5RFepssQM= -github.com/mitchellh/go-testing-interface v1.14.1 h1:jrgshOhYAUVNMAJiKbEu7EqAwgJJ2JqpQmpLJOu07cU= -github.com/mitchellh/go-testing-interface v1.14.1/go.mod h1:gfgS7OtZj6MA4U1UrDRp04twqAjfvlZyCfX3sDjEym8= -github.com/mitchellh/mapstructure v1.4.1 h1:CpVNEelQCZBooIPDn+AR3NpivK/TIKU8bDxdASFVQag= -github.com/mitchellh/mapstructure v1.4.1/go.mod h1:bFUtVrKA4DC2yAKiSyO/QUcy7e+RRV2QTWOzhPopBRo= -github.com/mitchellh/pointerstructure v1.2.0 h1:O+i9nHnXS3l/9Wu7r4NrEdwA2VFTicjUEN1uBnDo34A= -github.com/mitchellh/pointerstructure v1.2.0/go.mod h1:BRAsLI5zgXmw97Lf6s25bs8ohIXc3tViBH44KcwB2g4= -github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q= -github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd h1:TRLaZ9cD/w8PVh93nsPXa1VrQ6jlwL5oN8l14QlcNfg= -github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q= -github.com/modern-go/reflect2 v1.0.2 h1:xBagoLtFs94CBntxluKeaWgTMpvLxC4ur3nMaC9Gz0M= -github.com/modern-go/reflect2 v1.0.2/go.mod h1:yWuevngMOJpCy52FWWMvUC8ws7m/LJsjYzDa0/r8luk= -github.com/mostynb/zstdpool-freelist v0.0.0-20201229113212-927304c0c3b1 h1:mPMvm6X6tf4w8y7j9YIt6V9jfWhL6QlbEc7CCmeQlWk= -github.com/mostynb/zstdpool-freelist v0.0.0-20201229113212-927304c0c3b1/go.mod h1:ye2e/VUEtE2BHE+G/QcKkcLQVAEJoYRFj5VUOQatCRE= -github.com/mr-tron/base58 v1.2.0 h1:T/HDJBh4ZCPbU39/+c3rRvE0uKBQlU27+QI8LJ4t64o= -github.com/mr-tron/base58 v1.2.0/go.mod h1:BinMc/sQntlIE1frQmRFPUoPA1Zkr8VRgBdjWI2mNwc= -github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq1c1nUAm88MOHcQC9l5mIlSMApZMrHA= -github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ= -github.com/onsi/gomega v1.10.1 h1:o0+MgICZLuZ7xjH7Vx6zS/zcu93/BEp1VwkIW1mEXCE= -github.com/onsi/gomega v1.10.1/go.mod h1:iN09h71vgCQne3DLsj+A5owkum+a2tYe+TOCB1ybHNo= -github.com/pion/dtls/v2 v2.2.7 h1:cSUBsETxepsCSFSxC3mc/aDo14qQLMSL+O6IjG28yV8= -github.com/pion/dtls/v2 v2.2.7/go.mod h1:8WiMkebSHFD0T+dIU+UeBaoV7kDhOW5oDCzZ7WZ/F9s= -github.com/pion/logging v0.2.2 h1:M9+AIj/+pxNsDfAT64+MAVgJO0rsyLnoJKCqf//DoeY= -github.com/pion/logging v0.2.2/go.mod h1:k0/tDVsRCX2Mb2ZEmTqNa7CWsQPc+YYCB7Q+5pahoms= -github.com/pion/stun/v2 v2.0.0 h1:A5+wXKLAypxQri59+tmQKVs7+l6mMM+3d+eER9ifRU0= -github.com/pion/stun/v2 v2.0.0/go.mod h1:22qRSh08fSEttYUmJZGlriq9+03jtVmXNODgLccj8GQ= -github.com/pion/transport/v2 v2.2.1 h1:7qYnCBlpgSJNYMbLCKuSY9KbQdBFoETvPNETv0y4N7c= -github.com/pion/transport/v2 v2.2.1/go.mod h1:cXXWavvCnFF6McHTft3DWS9iic2Mftcz1Aq29pGcU5g= -github.com/pion/transport/v3 v3.0.1 h1:gDTlPJwROfSfz6QfSi0ZmeCSkFcnWWiiR9ES0ouANiM= -github.com/pion/transport/v3 v3.0.1/go.mod h1:UY7kiITrlMv7/IKgd5eTUcaahZx5oUN3l9SzK5f5xE0= -github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= -github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4= -github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= -github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= -github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= -github.com/prometheus/client_golang v1.20.5 h1:cxppBPuYhUnsO6yo/aoRol4L7q7UFfdm+bR9r+8l63Y= -github.com/prometheus/client_golang v1.20.5/go.mod h1:PIEt8X02hGcP8JWbeHyeZ53Y/jReSnHgO035n//V5WE= -github.com/prometheus/client_model v0.6.1 h1:ZKSh/rekM+n3CeS952MLRAdFwIKqeY8b62p8ais2e9E= -github.com/prometheus/client_model v0.6.1/go.mod h1:OrxVMOVHjw3lKMa8+x6HeMGkHMQyHDk9E3jmP2AmGiY= -github.com/prometheus/common v0.55.0 h1:KEi6DK7lXW/m7Ig5i47x0vRzuBsHuvJdi5ee6Y3G1dc= -github.com/prometheus/common v0.55.0/go.mod h1:2SECS4xJG1kd8XF9IcM1gMX6510RAEL65zxzNImwdc8= -github.com/prometheus/procfs v0.15.1 h1:YagwOFzUgYfKKHX6Dr+sHT7km/hxC76UB0learggepc= -github.com/prometheus/procfs v0.15.1/go.mod h1:fB45yRUv8NstnjriLhBQLuOUt+WW4BsoGhij/e3PBqk= -github.com/rogpeppe/go-internal v1.14.1 h1:UQB4HGPB6osV0SQTLymcB4TgvyWu6ZyliaW0tI/otEQ= -github.com/rogpeppe/go-internal v1.14.1/go.mod h1:MaRKkUm5W0goXpeCfT7UZI6fk/L7L7so1lCWt35ZSgc= -github.com/rs/cors v1.7.0 h1:+88SsELBHx5r+hZ8TCkggzSstaWNbDvThkVK8H6f9ik= -github.com/rs/cors v1.7.0/go.mod h1:gFx+x8UowdsKA9AchylcLynDq+nNFfI8FkUZdN/jGCU= -github.com/russross/blackfriday/v2 v2.1.0 h1:JIOH55/0cWyOuilr9/qlrm0BSXldqnqwMsf35Ld67mk= -github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM= -github.com/shirou/gopsutil v3.21.4-0.20210419000835-c7a38de76ee5+incompatible h1:Bn1aCHHRnjv4Bl16T8rcaFjYSrGrIZvpiGO6P3Q4GpU= -github.com/shirou/gopsutil v3.21.4-0.20210419000835-c7a38de76ee5+incompatible/go.mod h1:5b4v6he4MtMOwMlS0TUMTu2PcXUg8+E1lC7eC3UO/RA= -github.com/shopspring/decimal v1.3.1 h1:2Usl1nmF/WZucqkFZhnfFYxxxu8LG21F6nPQBE5gKV8= -github.com/shopspring/decimal v1.3.1/go.mod h1:DKyhrW/HYNuLGql+MJL6WCR6knT2jwCFRcu2hWCYk4o= -github.com/streamingfast/logging v0.0.0-20230608130331-f22c91403091/go.mod h1:VlduQ80JcGJSargkRU4Sg9Xo63wZD/l8A5NC/Uo1/uU= -github.com/streamingfast/logging v0.0.0-20250404134358-92b15d2fbd2e h1:qGVGDR2/bXLyR498un1hvhDQPUJ/m14JBRTJz+c67Bc= -github.com/streamingfast/logging v0.0.0-20250404134358-92b15d2fbd2e/go.mod h1:VlduQ80JcGJSargkRU4Sg9Xo63wZD/l8A5NC/Uo1/uU= -github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= -github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI= -github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= -github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U= -github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U= -github.com/supranational/blst v0.3.16 h1:bTDadT+3fK497EvLdWRQEjiGnUtzJ7jjIUMF0jqwYhE= -github.com/supranational/blst v0.3.16/go.mod h1:jZJtfjgudtNl4en1tzwPIV3KjUnQUvG3/j+w+fVonLw= -github.com/syndtr/goleveldb v1.0.1-0.20210819022825-2ae1ddf74ef7 h1:epCh84lMvA70Z7CTTCmYQn2CKbY8j86K7/FAIr141uY= -github.com/syndtr/goleveldb v1.0.1-0.20210819022825-2ae1ddf74ef7/go.mod h1:q4W45IWZaF22tdD+VEXcAWRA037jwmWEB5VWYORlTpc= -github.com/test-go/testify v1.1.4 h1:Tf9lntrKUMHiXQ07qBScBTSA0dhYQlu83hswqelv1iE= -github.com/test-go/testify v1.1.4/go.mod h1:rH7cfJo/47vWGdi4GPj16x3/t1xGOj2YxzmNQzk2ghU= -github.com/tklauser/go-sysconf v0.3.12 h1:0QaGUFOdQaIVdPgfITYzaTegZvdCjmYO52cSFAEVmqU= -github.com/tklauser/go-sysconf v0.3.12/go.mod h1:Ho14jnntGE1fpdOqQEEaiKRpvIavV0hSfmBq8nJbHYI= -github.com/tklauser/numcpus v0.6.1 h1:ng9scYS7az0Bk4OZLvrNXNSAO2Pxr1XXRAPyjhIx+Fk= -github.com/tklauser/numcpus v0.6.1/go.mod h1:1XfjsgE2zo8GVw7POkMbHENHzVg3GzmoZ9fESEdAacY= -github.com/urfave/cli/v2 v2.27.5 h1:WoHEJLdsXr6dDWoJgMq/CboDmyY/8HMMH1fTECbih+w= -github.com/urfave/cli/v2 v2.27.5/go.mod h1:3Sevf16NykTbInEnD0yKkjDAeZDS0A6bzhBH5hrMvTQ= -github.com/xrash/smetrics v0.0.0-20240521201337-686a1a2994c1 h1:gEOO8jv9F4OT7lGCjxCBTO/36wtF6j2nSip77qHd4x4= -github.com/xrash/smetrics v0.0.0-20240521201337-686a1a2994c1/go.mod h1:Ohn+xnUBiLI6FVj/9LpzZWtj1/D6lUovWYBkxHVV3aM= -github.com/yuin/goldmark v1.3.5/go.mod h1:mwnBkeHKe2W/ZEtQ+71ViKU8L12m81fl3OWwC1Zlc8k= -go.mongodb.org/mongo-driver/v2 v2.5.0 h1:yXUhImUjjAInNcpTcAlPHiT7bIXhshCTL3jVBkF3xaE= -go.mongodb.org/mongo-driver/v2 v2.5.0/go.mod h1:yOI9kBsufol30iFsl1slpdq1I0eHPzybRWdyYUs8K/0= -go.opentelemetry.io/auto/sdk v1.2.1 h1:jXsnJ4Lmnqd11kwkBV2LgLoFMZKizbCi5fNZ/ipaZ64= -go.opentelemetry.io/auto/sdk v1.2.1/go.mod h1:KRTj+aOaElaLi+wW1kO/DZRXwkF4C5xPbEe3ZiIhN7Y= -go.opentelemetry.io/otel v1.40.0 h1:oA5YeOcpRTXq6NN7frwmwFR0Cn3RhTVZvXsP4duvCms= -go.opentelemetry.io/otel v1.40.0/go.mod h1:IMb+uXZUKkMXdPddhwAHm6UfOwJyh4ct1ybIlV14J0g= -go.opentelemetry.io/otel/metric v1.40.0 h1:rcZe317KPftE2rstWIBitCdVp89A2HqjkxR3c11+p9g= -go.opentelemetry.io/otel/metric v1.40.0/go.mod h1:ib/crwQH7N3r5kfiBZQbwrTge743UDc7DTFVZrrXnqc= -go.opentelemetry.io/otel/sdk v1.40.0 h1:KHW/jUzgo6wsPh9At46+h4upjtccTmuZCFAc9OJ71f8= -go.opentelemetry.io/otel/sdk v1.40.0/go.mod h1:Ph7EFdYvxq72Y8Li9q8KebuYUr2KoeyHx0DRMKrYBUE= -go.opentelemetry.io/otel/trace v1.40.0 h1:WA4etStDttCSYuhwvEa8OP8I5EWu24lkOzp+ZYblVjw= -go.opentelemetry.io/otel/trace v1.40.0/go.mod h1:zeAhriXecNGP/s2SEG3+Y8X9ujcJOTqQ5RgdEJcawiA= -go.uber.org/atomic v1.4.0/go.mod h1:gD2HeocX3+yG+ygLZcrzQJaqmWj9AIm7n08wl/qW/PE= -go.uber.org/atomic v1.7.0/go.mod h1:fEN4uk6kAWBTFdckzkM89CLk9XfWZrxpCo0nPH17wJc= -go.uber.org/atomic v1.11.0 h1:ZvwS0R+56ePWxUNi+Atn9dWONBPp/AUETXlHW0DxSjE= -go.uber.org/atomic v1.11.0/go.mod h1:LUxbIzbOniOlMKjJjyPfpl4v+PKK2cNJn91OQbhoJI0= -go.uber.org/goleak v1.1.11/go.mod h1:cwTWslyiVhfpKIDGSZEM2HlOvcqm+tG4zioyIeLoqMQ= -go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto= -go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE= -go.uber.org/multierr v1.1.0/go.mod h1:wR5kodmAFQ0UK8QlbwjlSNy0Z68gJhDJUG5sjR94q/0= -go.uber.org/multierr v1.6.0/go.mod h1:cdWPpRnG4AhwMwsgIHip0KRBQjJy5kYEpYjJxpXp9iU= -go.uber.org/multierr v1.11.0 h1:blXXJkSxSSfBVBlC76pxqeO+LN3aDfLQo+309xJstO0= -go.uber.org/multierr v1.11.0/go.mod h1:20+QtiLqy0Nd6FdQB9TLXag12DsQkrbs3htMFfDN80Y= -go.uber.org/ratelimit v0.3.1 h1:K4qVE+byfv/B3tC+4nYWP7v/6SimcO7HzHekoMNBma0= -go.uber.org/ratelimit v0.3.1/go.mod h1:6euWsTB6U/Nb3X++xEUXA8ciPJvr19Q/0h1+oDcJhRk= -go.uber.org/zap v1.10.0/go.mod h1:vwi/ZaCAaUcBkycHslxD9B2zi4UTXhF60s6SWpuDF0Q= -go.uber.org/zap v1.21.0/go.mod h1:wjWOCqI0f2ZZrJF/UufIOkiC8ii6tm1iqIsLo76RfJw= -go.uber.org/zap v1.27.0 h1:aJMhYGrd5QSmlpLMr2MftRKl7t8J8PTZPA732ud/XR8= -go.uber.org/zap v1.27.0/go.mod h1:GB2qFLM7cTU87MWRP2mPIjqfIDnGu+VIO4V/SdhGo2E= -golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= -golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= -golang.org/x/crypto v0.0.0-20220214200702-86341886e292/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4= -golang.org/x/crypto v0.47.0 h1:V6e3FRj+n4dbpw86FJ8Fv7XVOql7TEwpHapKoMJ/GO8= -golang.org/x/crypto v0.47.0/go.mod h1:ff3Y9VzzKbwSSEzWqJsJVBnWmRwRSHt/6Op5n9bQc4A= -golang.org/x/exp v0.0.0-20230626212559-97b1e661b5df h1:UA2aFVmmsIlefxMk29Dp2juaUSth8Pyn3Tq5Y5mJGME= -golang.org/x/exp v0.0.0-20230626212559-97b1e661b5df/go.mod h1:FXUEEKJgO7OQYeo8N01OfiKP8RXMtf6e8aTskBGqWdc= -golang.org/x/lint v0.0.0-20190930215403-16217165b5de/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc= -golang.org/x/mod v0.4.2/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= -golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= -golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= -golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= -golang.org/x/net v0.0.0-20210405180319-a5a99cb37ef4/go.mod h1:p54w0d4576C0XHj96bSt6lcn1PtDYWL6XObtHCRCNQM= -golang.org/x/net v0.0.0-20211112202133-69e39bad7dc2/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y= -golang.org/x/net v0.49.0 h1:eeHFmOGUTtaaPSGNmjBKpbng9MulQsJURQUAfUwY++o= -golang.org/x/net v0.49.0/go.mod h1:/ysNB2EvaqvesRkuLAyjI1ycPZlQHM3q01F02UY/MV8= -golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= -golang.org/x/sync v0.0.0-20210220032951-036812b2e83c/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= -golang.org/x/sync v0.18.0 h1:kr88TuHDroi+UVf+0hZnirlk8o8T+4MrK6mr60WkH/I= -golang.org/x/sync v0.18.0/go.mod h1:9KTHXmSnoGruLpwFjVSX0lNNA75CykiMECbovNTZqGI= -golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= -golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20190916202348-b4ddaad3f8a3/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20210330210617-4fbd30eecc44/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20210423082822-04245dca01da/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20210510120138-977fb7262007/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.1.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.8.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.11.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.40.0 h1:DBZZqJ2Rkml6QMQsZywtnjnnGvHza6BTfYFWY9kjEWQ= -golang.org/x/sys v0.40.0/go.mod h1:OgkHotnGiDImocRcuBABYBEXf8A9a87e/uXjp9XT3ks= -golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo= -golang.org/x/term v0.39.0 h1:RclSuaJf32jOqZz74CkPA9qFuVTX7vhLlpfj/IGWlqY= -golang.org/x/term v0.39.0/go.mod h1:yxzUCTP/U+FzoxfdKmLaA0RV1WgE0VY7hXBwKtY/4ww= -golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= -golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= -golang.org/x/text v0.3.6/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= -golang.org/x/text v0.33.0 h1:B3njUFyqtHDUI5jMn1YIr5B0IE2U0qck04r6d4KPAxE= -golang.org/x/text v0.33.0/go.mod h1:LuMebE6+rBincTi9+xWTY8TztLzKHc/9C1uBCG27+q8= -golang.org/x/time v0.11.0 h1:/bpjEDfN9tkoN/ryeYHnv5hcMlc8ncjMcM4XBk5NWV0= -golang.org/x/time v0.11.0/go.mod h1:CDIdPxbZBQxdj6cxyCIdrNogrJKMJ7pr37NYpMcMDSg= -golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= -golang.org/x/tools v0.0.0-20190311212946-11955173bddd/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs= -golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= -golang.org/x/tools v0.1.5/go.mod h1:o0xws9oXOQQZyjljx8fwUC0k7L1pTE6eaCbjGeHmOkk= -golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= -golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= -golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 h1:go1bK/D/BFZV2I8cIQd1NKEZ+0owSTG1fDTci4IqFcE= -golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= -google.golang.org/protobuf v1.36.11 h1:fV6ZwhNocDyBLK0dj+fg8ektcVegBBuEolpbTQyBNVE= -google.golang.org/protobuf v1.36.11/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco= -gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= -gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= -gopkg.in/natefinch/lumberjack.v2 v2.2.1 h1:bBRl1b0OH9s/DuPhuXpNl+VtCaJXFZ5/uEFST95x9zc= -gopkg.in/natefinch/lumberjack.v2 v2.2.1/go.mod h1:YD8tP3GAjkrDg1eZH7EGmyESg/lsYskCTPBJVb9jqSc= -gopkg.in/yaml.v2 v2.2.8/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= -gopkg.in/yaml.v2 v2.4.0 h1:D8xgwECY7CYvx+Y2n4sBz93Jn9JRvxdiyyo8CTfuKaY= -gopkg.in/yaml.v2 v2.4.0/go.mod h1:RDklbk79AGWmwhnvt/jBztapEOGDOx6ZbXqjP6csGnQ= -gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= -gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= -gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= -gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= diff --git a/harnesses/metadata-coverage/.gitignore b/harnesses/metadata-coverage/.gitignore deleted file mode 100644 index ec3fb764..00000000 --- a/harnesses/metadata-coverage/.gitignore +++ /dev/null @@ -1,34 +0,0 @@ -# Environment variables (contains secrets) -.env - -# Build artifacts -benchmark -*.exe -*.dll -*.so -*.dylib - -# Test binaries -*.test -*.out - -# Go workspace file -go.work - -# IDE -.vscode/ -.idea/ -*.swp -*.swo -*~ - -# OS -.DS_Store -Thumbs.db - -# Logs -*.log - -# Temporary files -tmp/ -temp/ diff --git a/harnesses/metadata-coverage/Dockerfile b/harnesses/metadata-coverage/Dockerfile deleted file mode 100644 index 74ca8f0f..00000000 --- a/harnesses/metadata-coverage/Dockerfile +++ /dev/null @@ -1,36 +0,0 @@ -# Build stage -FROM golang:1.24-alpine AS builder - -WORKDIR /app - -# Install dependencies -RUN apk add --no-cache git - -# Copy go mod files -COPY go.mod go.sum ./ -RUN go mod download - -# Copy source code -COPY . . - -# Build the binary -RUN CGO_ENABLED=0 GOOS=linux go build -o /app/monitor ./cmd/script - -# Runtime stage -FROM debian:bookworm-slim - -WORKDIR /app - -# Install runtime dependencies -RUN apt-get update && apt-get install -y \ - ca-certificates \ - && rm -rf /var/lib/apt/lists/* - -# Copy binary from builder -COPY --from=builder /app/monitor /app/monitor - -# Expose metrics port -EXPOSE 2112 - -# Run the monitor -CMD ["/app/monitor"] diff --git a/harnesses/metadata-coverage/README.md b/harnesses/metadata-coverage/README.md deleted file mode 100644 index 6c02e1be..00000000 --- a/harnesses/metadata-coverage/README.md +++ /dev/null @@ -1,49 +0,0 @@ -# Harness · metadata-coverage - -> Token-metadata field-coverage monitor that produces the `metadata_coverage_*` metrics consumed by [`benchmarks/metadata-coverage.yml`](../../benchmarks/metadata-coverage.yml). One Go binary, one Railway service, one benchmark. - -**Bench**: [№ 004 · Token Metadata Coverage](../../benchmarks/metadata-coverage.yml) - -## How it works - -Two goroutines inside one binary: - -1. **Pulse V2 feeder** subscribes to Mobula's Pulse V2 WebSocket on Solana / BNB / Base and pushes freshly-launched launchpad tokens (pump.fun, Meteora DBC, Four.meme, Raydium CPMM, Zora, BaseApp, Bags, Moonshot) onto an in-memory queue. -2. **Metadata coverage worker** consumes the queue, queries each aggregator's metadata endpoint, and records whether four canonical fields (`logo`, `description`, `twitter`, `website`) are returned. - -Pulse itself emits no benchmark metric. It is only used to discover fresh tokens. - -**Tracked aggregators**: Mobula · Codex · Jupiter (Solana only) -**Tracked fields**: logo · description · twitter · website - -## Metrics produced - -``` -metadata_coverage_checks_total{provider, chain, field, region} counter -metadata_coverage_success_total{provider, chain, field, region} counter -metadata_api_latency_milliseconds{provider, chain, region} histogram -``` - -The bench page renders coverage % as `success_total / checks_total × 100`. - -## Run locally - -```bash -cp .env.example .env -# Fill in MOBULA_API_KEY, DEFINED_SESSION_COOKIE -go run ./cmd/script/ -``` - -`/metrics` exposed on `http://localhost:2112/metrics`. - -## Environment variables - -| Var | Description | Required | -| --- | --- | --- | -| `MOBULA_API_KEY` | Mobula API key (Pulse V2 + token details) | yes | -| `DEFINED_SESSION_COOKIE` | Defined.fi session cookie (Codex GraphQL via JWT mint) | yes | -| `MONITOR_REGION` | Label written on every metric | recommended | - -## License - -MIT, same as the rest of OpenChainBench. diff --git a/harnesses/metadata-coverage/cmd/script/cleanup.go b/harnesses/metadata-coverage/cmd/script/cleanup.go deleted file mode 100644 index fb1d5883..00000000 --- a/harnesses/metadata-coverage/cmd/script/cleanup.go +++ /dev/null @@ -1,137 +0,0 @@ -package main - -import ( - "fmt" - "log" - "net/http" - "os" - "path/filepath" - "time" -) - -// setupCleanupEndpoint adds an admin endpoint for selective data cleanup -func setupCleanupEndpoint(mux *http.ServeMux) { - adminToken := os.Getenv("ADMIN_CLEANUP_TOKEN") - if adminToken == "" { - log.Println("[CLEANUP] Warning: ADMIN_CLEANUP_TOKEN not set - cleanup endpoint will be disabled") - return - } - - mux.HandleFunc("/admin/cleanup", func(w http.ResponseWriter, r *http.Request) { - // Only allow DELETE method - if r.Method != "DELETE" { - http.Error(w, "Method not allowed - use DELETE", http.StatusMethodNotAllowed) - return - } - - // Check authorization - authHeader := r.Header.Get("Authorization") - expectedAuth := "Bearer " + adminToken - if authHeader != expectedAuth { - log.Printf("[CLEANUP] Unauthorized cleanup attempt from %s", r.RemoteAddr) - http.Error(w, "Unauthorized", http.StatusUnauthorized) - return - } - - // Parse the 'before' timestamp parameter - beforeStr := r.URL.Query().Get("before") - if beforeStr == "" { - http.Error(w, "Missing 'before' parameter (use RFC3339 format: 2026-04-10T00:00:00Z)", http.StatusBadRequest) - return - } - - before, err := time.Parse(time.RFC3339, beforeStr) - if err != nil { - http.Error(w, fmt.Sprintf("Invalid timestamp format: %v (use RFC3339: 2026-04-10T00:00:00Z)", err), http.StatusBadRequest) - return - } - - // Get data paths from environment or use defaults - prometheusPath := os.Getenv("PROMETHEUS_DATA_PATH") - if prometheusPath == "" { - prometheusPath = "/data/prometheus" - } - - grafanaPath := os.Getenv("GRAFANA_DATA_PATH") - if grafanaPath == "" { - grafanaPath = "/data/grafana" - } - - log.Printf("[CLEANUP] Starting cleanup: deleting data before %s", before.Format(time.RFC3339)) - log.Printf("[CLEANUP] Prometheus path: %s", prometheusPath) - log.Printf("[CLEANUP] Grafana path: %s", grafanaPath) - - totalDeleted := 0 - - // Cleanup Prometheus data - if _, err := os.Stat(prometheusPath); err == nil { - deleted, err := cleanupDirectory(prometheusPath, before) - if err != nil { - log.Printf("[CLEANUP] Error cleaning Prometheus data: %v", err) - http.Error(w, fmt.Sprintf("Prometheus cleanup failed: %v", err), http.StatusInternalServerError) - return - } - totalDeleted += deleted - log.Printf("[CLEANUP] Deleted %d items from Prometheus", deleted) - } else { - log.Printf("[CLEANUP] Prometheus path not found: %s", prometheusPath) - } - - // Cleanup Grafana data - if _, err := os.Stat(grafanaPath); err == nil { - deleted, err := cleanupDirectory(grafanaPath, before) - if err != nil { - log.Printf("[CLEANUP] Error cleaning Grafana data: %v", err) - http.Error(w, fmt.Sprintf("Grafana cleanup failed: %v", err), http.StatusInternalServerError) - return - } - totalDeleted += deleted - log.Printf("[CLEANUP] Deleted %d items from Grafana", deleted) - } else { - log.Printf("[CLEANUP] Grafana path not found: %s", grafanaPath) - } - - response := fmt.Sprintf("✅ Cleanup complete\n\nDeleted %d files/directories before %s\n\nPaths cleaned:\n- %s\n- %s\n", - totalDeleted, before.Format(time.RFC3339), prometheusPath, grafanaPath) - - log.Printf("[CLEANUP] Cleanup complete: %d items deleted", totalDeleted) - w.Header().Set("Content-Type", "text/plain") - w.WriteHeader(http.StatusOK) - fmt.Fprint(w, response) - }) - - log.Printf("[CLEANUP] Admin cleanup endpoint enabled at /admin/cleanup") -} - -// cleanupDirectory removes files and directories older than the specified time -func cleanupDirectory(dir string, before time.Time) (int, error) { - deleted := 0 - - entries, err := os.ReadDir(dir) - if err != nil { - return 0, fmt.Errorf("failed to read directory: %w", err) - } - - for _, entry := range entries { - path := filepath.Join(dir, entry.Name()) - info, err := entry.Info() - if err != nil { - log.Printf("[CLEANUP] Warning: failed to get info for %s: %v", path, err) - continue - } - - // Check if modification time is before the cutoff - if info.ModTime().Before(before) { - log.Printf("[CLEANUP] Deleting: %s (modified: %s)", path, info.ModTime().Format(time.RFC3339)) - - if err := os.RemoveAll(path); err != nil { - log.Printf("[CLEANUP] Warning: failed to delete %s: %v", path, err) - continue - } - - deleted++ - } - } - - return deleted, nil -} diff --git a/harnesses/metadata-coverage/cmd/script/codex_rest_monitor.go b/harnesses/metadata-coverage/cmd/script/codex_rest_monitor.go deleted file mode 100644 index f75f0c00..00000000 --- a/harnesses/metadata-coverage/cmd/script/codex_rest_monitor.go +++ /dev/null @@ -1,245 +0,0 @@ -package main - -import ( - "bytes" - "encoding/json" - "fmt" - "io" - "log" - "net/http" - "strings" - "sync" - "time" -) - -const ( - codexRESTBaseURL = "https://graph.codex.io/graphql" -) - -// Track rate limiting to avoid spamming JWT generation -var ( - codexRESTRateLimitedUntil time.Time - codexRESTRateLimitMutex sync.RWMutex -) - -// Chains for REST monitoring - aligned with all monitors -var codexRESTChains = []struct { - networkID int - chainName string - poolAddress string -}{ - {1399811149, "solana", "7qbRF6YsyGuLUVs6Y1q64bdVrfe4ZcUUz1JRdoVNUJnm"}, // SOL/USDC Raydium - {8453, "base", "0x4c36388be6f416a29c8d8eee81c771ce6be14b18"}, // WETH/USDC Base - {56, "bnb", "0x58f876857a02d6762e0101bb5c46a8c1ed44dc16"}, // WBNB/BUSD PancakeSwap -} - -type CodexGraphQLRequest struct { - Query string `json:"query"` - Variables map[string]interface{} `json:"variables"` -} - -type CodexGraphQLResponse struct { - Data map[string]interface{} `json:"data"` - Errors []struct { - Message string `json:"message"` - } `json:"errors"` -} - -// callCodexGraphQLAPI makes a GraphQL query to Codex API -func callCodexGraphQLAPI(apiKey string, poolAddress string, networkID int, chainName string) (float64, int, error) { - // Create HTTP client with proxy support (forces new connection for IP rotation) - client := getProxyHTTPClient() - - // Build GraphQL query - filterPairs is reliable and works for all chains - // This query filters pairs by network and returns one result to measure latency - query := ` - query FilterPairs($networkId: [Int!]) { - filterPairs(filters: { network: $networkId }, limit: 1) { - results { - pair { - address - token0 - token1 - } - } - } - } - ` - - // Build request body with variables - reqBody := CodexGraphQLRequest{ - Query: query, - Variables: map[string]interface{}{ - "networkId": []int{networkID}, - }, - } - - bodyBytes, err := json.Marshal(reqBody) - if err != nil { - return 0, 0, fmt.Errorf("failed to marshal request: %w", err) - } - - // Build request - req, err := http.NewRequest("POST", codexRESTBaseURL, bytes.NewBuffer(bodyBytes)) - if err != nil { - return 0, 0, fmt.Errorf("failed to create request: %w", err) - } - - // Add headers - req.Header.Set("Authorization", fmt.Sprintf("Bearer %s", apiKey)) - req.Header.Set("Content-Type", "application/json") - - // Measure latency - startTime := time.Now() - resp, err := client.Do(req) - latencyMs := float64(time.Since(startTime).Milliseconds()) - - if err != nil { - return latencyMs, 0, fmt.Errorf("request failed: %w", err) - } - defer resp.Body.Close() - - // Read response body - body, _ := io.ReadAll(resp.Body) - - // Try to parse response - var graphqlResp CodexGraphQLResponse - if err := json.Unmarshal(body, &graphqlResp); err != nil { - log.Printf("[CODEX-REST][%s] Response parse warning: %v (status: %d)", chainName, err, resp.StatusCode) - } - - // Check for GraphQL errors - if len(graphqlResp.Errors) > 0 { - log.Printf("[CODEX-REST][%s] GraphQL errors: %v", chainName, graphqlResp.Errors[0].Message) - - // Check if it's an authentication error - if graphqlResp.Errors[0].Message == "User is not authenticated" { - return latencyMs, resp.StatusCode, fmt.Errorf("authentication error: %s", graphqlResp.Errors[0].Message) - } - } - - return latencyMs, resp.StatusCode, nil -} - -// monitorCodexREST continuously monitors Codex GraphQL API latency -func monitorCodexREST(config *Config, stopChan <-chan struct{}) { - fmt.Println("Starting Codex REST API monitor...") - fmt.Printf(" Monitoring %d chains with 20s interval\n", len(codexRESTChains)) - fmt.Printf(" Endpoint: POST /graphql (GraphQL)\n") - fmt.Println() - - if config.DefinedSessionCookie == "" { - fmt.Println("DEFINED_SESSION_COOKIE not set in .env file. Skipping Codex REST monitor.") - return - } - - // Create ticker for 20 second intervals - ticker := time.NewTicker(20 * time.Second) - defer ticker.Stop() - - // Run once immediately - performCodexRESTChecks(config) - - // Then run every 20 seconds - for { - select { - case <-stopChan: - fmt.Println("Codex REST monitor stopped") - return - case <-ticker.C: - performCodexRESTChecks(config) - } - } -} - -// performCodexRESTChecks performs GraphQL API calls to all chains -func performCodexRESTChecks(config *Config) { - timestamp := time.Now().UTC().Format("2006-01-02 15:04:05") - - // Check if we're still rate limited - codexRESTRateLimitMutex.RLock() - if time.Now().Before(codexRESTRateLimitedUntil) { - waitTime := time.Until(codexRESTRateLimitedUntil) - codexRESTRateLimitMutex.RUnlock() - fmt.Printf("[CODEX-REST] Skipping checks - rate limited for another %v\n", waitTime.Round(time.Second)) - return - } - codexRESTRateLimitMutex.RUnlock() - - // Generate JWT token from session cookie - jwtToken, err := GetDefinedJWTToken(config.DefinedSessionCookie) - if err != nil { - fmt.Printf("[CODEX-REST] Failed to get JWT token: %v\n", err) - - // If rate limited, stop trying for 10 minutes - if strings.Contains(err.Error(), "rate limited (429)") || strings.Contains(err.Error(), "too many token requests") { - codexRESTRateLimitMutex.Lock() - codexRESTRateLimitedUntil = time.Now().Add(10 * time.Minute) - codexRESTRateLimitMutex.Unlock() - fmt.Printf("[CODEX-REST] ⏱️ Rate limited - pausing checks for 10 minutes\n") - } - return - } - - for _, chain := range codexRESTChains { - latencyMs, statusCode, err := callCodexGraphQLAPI( - jwtToken, - chain.poolAddress, - chain.networkID, - chain.chainName, - ) - - if err != nil { - // Check if it's an auth error - if err.Error() == "authentication error: User is not authenticated" { - fmt.Println("[CODEX-REST] Authentication error - invalidating JWT cache") - InvalidateTokenCache() - } - - // Record error - errorType := "request_error" - if statusCode >= 500 { - errorType = "server_error" - } else if statusCode >= 400 { - errorType = "client_error" - } else if statusCode == 0 { - errorType = "timeout_error" - } - - RecordRESTError("codex", "graphql", chain.chainName, errorType, config.MonitorRegion) - - fmt.Printf("[CODEX-REST][%s][%s] ERROR | Latency: %.0fms | Status: %d | Error: %v\n", - timestamp, - chain.chainName, - latencyMs, - statusCode, - err, - ) - continue - } - - // Record successful latency measurement - RecordRESTLatency("codex", "graphql", chain.chainName, latencyMs, statusCode, config.MonitorRegion) - - // Log the result - statusEmoji := "✓" - if statusCode >= 400 { - statusEmoji = "✗" - } else if statusCode >= 300 { - statusEmoji = "⚠" - } - - fmt.Printf("[CODEX-REST][%s][%s] %s | Latency: %.0fms | Status: %d\n", - timestamp, - chain.chainName, - statusEmoji, - latencyMs, - statusCode, - ) - } -} - -// runCodexRESTMonitor is the entry point for the Codex REST monitor -func runCodexRESTMonitor(config *Config, stopChan <-chan struct{}) { - monitorCodexREST(config, stopChan) -} diff --git a/harnesses/metadata-coverage/cmd/script/config.go b/harnesses/metadata-coverage/cmd/script/config.go deleted file mode 100644 index a3fd5114..00000000 --- a/harnesses/metadata-coverage/cmd/script/config.go +++ /dev/null @@ -1,85 +0,0 @@ -package main - -import ( - "bufio" - "fmt" - "os" - "strings" -) - -type Config struct { - CoinGeckoAPIKey string - MobulaAPIKey string - DefinedSessionCookie string - MonitorRegion string // Deployment region: us-west, us-east, singapore, etc. - MobulaWSURL string // Mobula fast-trade WebSocket endpoint (allows staging to use EU-specific cluster) -} - -func loadEnv() (*Config, error) { - config := &Config{} - - // First, try to load from environment variables (for production/Railway) - config.CoinGeckoAPIKey = strings.TrimSpace(os.Getenv("COINGECKO_API_KEY")) - config.MobulaAPIKey = strings.TrimSpace(os.Getenv("MOBULA_API_KEY")) - config.DefinedSessionCookie = strings.TrimSpace(os.Getenv("DEFINED_SESSION_COOKIE")) - config.MonitorRegion = strings.TrimSpace(os.Getenv("MONITOR_REGION")) - config.MobulaWSURL = strings.TrimSpace(os.Getenv("MOBULA_WS_URL")) - - // Default to "unknown" if not set - if config.MonitorRegion == "" { - config.MonitorRegion = "unknown" - } - - // Default Mobula WS endpoint (global). Staging can override to wss://api-prod-eu.mobula.io - if config.MobulaWSURL == "" { - config.MobulaWSURL = "wss://api.mobula.io" - } - - // If all env vars are set, return early (production mode) - if config.CoinGeckoAPIKey != "" || config.MobulaAPIKey != "" || config.DefinedSessionCookie != "" { - return config, nil - } - - // Otherwise, try to load from .env file (for local development) - file, err := os.Open(".env") - if err != nil { - // If no .env file and no env vars, that's OK - services will just be skipped - return config, nil - } - defer file.Close() - - scanner := bufio.NewScanner(file) - for scanner.Scan() { - line := strings.TrimSpace(scanner.Text()) - if line == "" || strings.HasPrefix(line, "#") { - continue - } - - parts := strings.SplitN(line, "=", 2) - if len(parts) != 2 { - continue - } - - key, value := strings.TrimSpace(parts[0]), strings.TrimSpace(parts[1]) - switch key { - case "COINGECKO_API_KEY": - if config.CoinGeckoAPIKey == "" { - config.CoinGeckoAPIKey = value - } - case "MOBULA_API_KEY": - if config.MobulaAPIKey == "" { - config.MobulaAPIKey = value - } - case "DEFINED_SESSION_COOKIE": - if config.DefinedSessionCookie == "" { - config.DefinedSessionCookie = value - } - } - } - - if err := scanner.Err(); err != nil { - return nil, fmt.Errorf("error reading .env file: %w", err) - } - - return config, nil -} diff --git a/harnesses/metadata-coverage/cmd/script/defined_auth.go b/harnesses/metadata-coverage/cmd/script/defined_auth.go deleted file mode 100644 index 716f05f0..00000000 --- a/harnesses/metadata-coverage/cmd/script/defined_auth.go +++ /dev/null @@ -1,197 +0,0 @@ -package main - -import ( - "bytes" - "encoding/base64" - "encoding/json" - "fmt" - "io" - "net/http" - "strings" - "sync" - "time" -) - -type DefinedTokenResponse struct { - Data struct { - CreateApiTokens []struct { - Token string `json:"token"` - } `json:"createApiTokens"` - } `json:"data"` -} - -// JWT token cache to avoid rate limiting -type tokenCache struct { - mu sync.RWMutex - token string - expiresAt time.Time - lastRefresh time.Time -} - -var globalTokenCache = &tokenCache{} - -// decodeJWTExpiration extracts the expiration time from a JWT token -func decodeJWTExpiration(token string) (time.Time, error) { - parts := strings.Split(token, ".") - if len(parts) != 3 { - return time.Time{}, fmt.Errorf("invalid JWT format") - } - - // Decode payload (second part) - payload, err := base64.RawURLEncoding.DecodeString(parts[1]) - if err != nil { - return time.Time{}, fmt.Errorf("failed to decode JWT payload: %w", err) - } - - var claims struct { - Exp int64 `json:"exp"` - } - if err := json.Unmarshal(payload, &claims); err != nil { - return time.Time{}, fmt.Errorf("failed to unmarshal JWT claims: %w", err) - } - - if claims.Exp == 0 { - return time.Time{}, fmt.Errorf("no expiration in JWT") - } - - return time.Unix(claims.Exp, 0), nil -} - -// GetDefinedJWTToken returns a cached JWT token or generates a new one if expired -func GetDefinedJWTToken(sessionCookie string) (string, error) { - globalTokenCache.mu.RLock() - - // Check if we have a valid cached token - // Renew 1 hour before expiration to be safe - if globalTokenCache.token != "" && time.Now().Before(globalTokenCache.expiresAt.Add(-1*time.Hour)) { - token := globalTokenCache.token - globalTokenCache.mu.RUnlock() - return token, nil - } - globalTokenCache.mu.RUnlock() - - // Need to refresh token - globalTokenCache.mu.Lock() - defer globalTokenCache.mu.Unlock() - - // Double-check after acquiring write lock - if globalTokenCache.token != "" && time.Now().Before(globalTokenCache.expiresAt.Add(-1*time.Hour)) { - return globalTokenCache.token, nil - } - - // Generate new token - token, err := generateDefinedJWTToken(sessionCookie) - if err != nil { - return "", err - } - - // Decode expiration from token - expiresAt, err := decodeJWTExpiration(token) - if err != nil { - fmt.Printf("[DEFINED-AUTH] Warning: Could not decode token expiration: %v. Will cache for 24h.\n", err) - expiresAt = time.Now().Add(24 * time.Hour) - } - - // Cache the token - globalTokenCache.token = token - globalTokenCache.expiresAt = expiresAt - globalTokenCache.lastRefresh = time.Now() - - timeUntilExpiry := time.Until(expiresAt) - fmt.Printf("[DEFINED-AUTH] JWT token refreshed. Expires in %.1fh (at %s)\n", - timeUntilExpiry.Hours(), expiresAt.Format("2006-01-02 15:04:05")) - - return token, nil -} - -// generateDefinedJWTToken generates a new JWT token from Defined.fi session cookie -func generateDefinedJWTToken(sessionCookie string) (string, error) { - fmt.Println("[DEFINED-AUTH] Generating new JWT token from Defined.fi (local)...") - fmt.Println("[DEFINED-AUTH] Creating new HTTP client with fresh TCP connection (no keepalive)") - - // Create a new HTTP client with fresh connection for each request. - // CRITICAL: route through HTTP_PROXY/HTTPS_PROXY (webshare rotating proxy) - // so each JWT mint hits a fresh IP. Direct from the container IP gets us - // stuck on Vercel's bot ban (429 loop) when the container restarts often. - transport := &http.Transport{ - DisableKeepAlives: true, - MaxIdleConnsPerHost: 0, - Proxy: http.ProxyFromEnvironment, - } - client := &http.Client{ - Timeout: 10 * time.Second, - Transport: transport, - } - - reqBody := map[string]interface{}{ - "operationName": "CreateApiToken", - "query": "mutation CreateApiToken { createApiTokens(input: { count: 1 }) { token } }", - "variables": map[string]interface{}{}, - } - - bodyBytes, _ := json.Marshal(reqBody) - req, _ := http.NewRequest("POST", "https://www.defined.fi/api", bytes.NewBuffer(bodyBytes)) - - req.Header.Set("Accept", "application/json") - req.Header.Set("Accept-Language", "en-US,en;q=0.9") - req.Header.Set("Content-Type", "application/json") - req.Header.Set("Origin", "https://www.defined.fi") - req.Header.Set("Referer", "https://www.defined.fi/") - req.Header.Set("User-Agent", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36") - req.Header.Set("sec-ch-ua", `"Not_A Brand";v="8", "Chromium";v="131", "Google Chrome";v="131"`) - req.Header.Set("sec-ch-ua-mobile", "?0") - req.Header.Set("sec-ch-ua-platform", `"macOS"`) - req.Header.Set("sec-fetch-dest", "empty") - req.Header.Set("sec-fetch-mode", "cors") - req.Header.Set("sec-fetch-site", "same-origin") - req.AddCookie(&http.Cookie{Name: "session", Value: sessionCookie}) - - fmt.Println("[DEFINED-AUTH] Sending POST request to https://www.defined.fi/api...") - resp, err := client.Do(req) - if err != nil { - fmt.Printf("[DEFINED-AUTH] ❌ Request failed: %v\n", err) - return "", fmt.Errorf("request failed: %w", err) - } - defer resp.Body.Close() - - respBody, _ := io.ReadAll(resp.Body) - fmt.Printf("[DEFINED-AUTH] Response status: %d\n", resp.StatusCode) - - if resp.StatusCode == 429 { - // Parse retry-after header if available - retryAfter := resp.Header.Get("Retry-After") - fmt.Printf("[DEFINED-AUTH] ⚠ Rate limited! Retry-After: %s\n", retryAfter) - if retryAfter != "" { - return "", fmt.Errorf("rate limited (429), retry after: %s", retryAfter) - } - return "", fmt.Errorf("rate limited (429), too many token requests - will retry later") - } - - if resp.StatusCode != 200 { - fmt.Printf("[DEFINED-AUTH] ❌ Unexpected status %d: %s\n", resp.StatusCode, string(respBody[:min(len(respBody), 100)])) - return "", fmt.Errorf("unexpected status %d: %s", resp.StatusCode, string(respBody[:min(len(respBody), 100)])) - } - - var tokenResp DefinedTokenResponse - if err := json.Unmarshal(respBody, &tokenResp); err != nil { - fmt.Printf("[DEFINED-AUTH] ❌ Failed to decode response: %v\n", err) - return "", fmt.Errorf("failed to decode: %w", err) - } - - if len(tokenResp.Data.CreateApiTokens) == 0 { - fmt.Println("[DEFINED-AUTH] ❌ No token in response") - return "", fmt.Errorf("no token returned") - } - - fmt.Printf("[DEFINED-AUTH] ✅ JWT token generated successfully (length: %d)\n", len(tokenResp.Data.CreateApiTokens[0].Token)) - return tokenResp.Data.CreateApiTokens[0].Token, nil -} - -// InvalidateTokenCache forces a token refresh on next request -func InvalidateTokenCache() { - globalTokenCache.mu.Lock() - defer globalTokenCache.mu.Unlock() - globalTokenCache.token = "" - globalTokenCache.expiresAt = time.Time{} - fmt.Println("[DEFINED-AUTH] Token cache invalidated - will refresh on next request") -} diff --git a/harnesses/metadata-coverage/cmd/script/geckoterminal_monitor.go b/harnesses/metadata-coverage/cmd/script/geckoterminal_monitor.go deleted file mode 100644 index c19e7edd..00000000 --- a/harnesses/metadata-coverage/cmd/script/geckoterminal_monitor.go +++ /dev/null @@ -1,263 +0,0 @@ -package main - -import ( - "encoding/json" - "fmt" - "log" - "sync" - "time" - - "github.com/gorilla/websocket" -) - -// ============================================================================ -// GeckoTerminal WebSocket Monitor -// ============================================================================ - -const ( - geckoWSURL = "wss://cables.geckoterminal.com/cable" - geckoOrigin = "https://www.geckoterminal.com" - geckoUserAgent = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36" -) - -// GeckoTerminal pools (pool_id extracted via reverse engineering) -var geckoTerminalPools = []struct { - Name string - Network string - PoolID string - Chain string -}{ - { - Name: "SOL/USDC Raydium", - Network: "solana", - PoolID: "162715608", - Chain: "solana", - }, - { - Name: "WETH/USDC Base", - Network: "base", - PoolID: "162840764", - Chain: "base", - }, - { - Name: "WBNB/BUSD PancakeSwap", - Network: "bsc", - PoolID: "24", - Chain: "bnb", - }, -} - -// ActionCable message structures -type GeckoActionCableMessage struct { - Type string `json:"type,omitempty"` - Command string `json:"command,omitempty"` - Identifier string `json:"identifier,omitempty"` - Message json.RawMessage `json:"message,omitempty"` -} - -type GeckoChannelIdentifier struct { - Channel string `json:"channel"` - PoolID string `json:"pool_id,omitempty"` -} - -// Swap data from SwapChannel -type GeckoSwapData struct { - Data struct { - BlockTimestamp int64 `json:"block_timestamp"` // On-chain timestamp (ms) - TxHash string `json:"tx_hash"` - // Other fields available but not needed for head lag - } `json:"data"` - Type string `json:"type"` // "newSwap" -} - -func runGeckoTerminalHeadLagMonitor(config *Config, stopChan <-chan struct{}, wg *sync.WaitGroup) { - defer wg.Done() - - fmt.Println("[HEAD-LAG][GECKO] Starting WebSocket monitor...") - - reconnectDelay := 5 * time.Second - maxReconnectDelay := 60 * time.Second - - for { - select { - case <-stopChan: - fmt.Println("[HEAD-LAG][GECKO] Monitor stopped") - return - default: - err := connectAndMonitorGecko(config, stopChan) - if err != nil { - log.Printf("[HEAD-LAG][GECKO] Connection error: %v. Reconnecting in %v...", err, reconnectDelay) - - select { - case <-stopChan: - return - case <-time.After(reconnectDelay): - reconnectDelay = reconnectDelay * 2 - if reconnectDelay > maxReconnectDelay { - reconnectDelay = maxReconnectDelay - } - } - } else { - reconnectDelay = 5 * time.Second - } - } - } -} - -func connectAndMonitorGecko(config *Config, stopChan <-chan struct{}) error { - headers := map[string][]string{ - "Origin": {geckoOrigin}, - "User-Agent": {geckoUserAgent}, - } - - conn, _, err := getProxyDialer().Dial(geckoWSURL, headers) - if err != nil { - return fmt.Errorf("dial failed: %w", err) - } - defer conn.Close() - - // Channel for messages - done := make(chan struct{}) - - // Read messages goroutine - go func() { - defer close(done) - for { - _, message, err := conn.ReadMessage() - if err != nil { - return - } - - handleGeckoMessage(config, conn, message) - } - }() - - // Wait for welcome message - time.Sleep(2 * time.Second) - - // Subscribe to SwapChannel for all monitored pools - for _, pool := range geckoTerminalPools { - subscribeToGeckoSwapChannel(conn, pool.PoolID, pool.Name) - time.Sleep(100 * time.Millisecond) - } - - fmt.Printf("[HEAD-LAG][GECKO] Subscribed to %d pools\n", len(geckoTerminalPools)) - - // Heartbeat ticker - pingTicker := time.NewTicker(25 * time.Second) - defer pingTicker.Stop() - - // Read messages - for { - select { - case <-stopChan: - return nil - case <-done: - return fmt.Errorf("connection closed by server") - case <-pingTicker.C: - // Server sends pings, we respond with pongs (handled in handleGeckoMessage) - conn.SetWriteDeadline(time.Now().Add(10 * time.Second)) - } - } -} - -func handleGeckoMessage(config *Config, conn *websocket.Conn, message []byte) { - var msg GeckoActionCableMessage - if err := json.Unmarshal(message, &msg); err != nil { - return - } - - switch msg.Type { - case "welcome": - // Connection established - - case "ping": - // Respond to ping with pong - pong := GeckoActionCableMessage{ - Type: "pong", - } - conn.WriteJSON(pong) - - case "confirm_subscription": - // Subscription confirmed - - case "reject_subscription": - log.Printf("[HEAD-LAG][GECKO] Subscription rejected: %s", msg.Identifier) - - default: - // Handle data messages - if msg.Message != nil { - handleGeckoDataMessage(config, msg.Identifier, msg.Message) - } - } -} - -func handleGeckoDataMessage(config *Config, identifier string, message json.RawMessage) { - // Parse swap data - var swapData GeckoSwapData - if err := json.Unmarshal(message, &swapData); err != nil { - return - } - - if swapData.Type != "newSwap" { - return - } - - // Extract channel info to get pool - var channelIdent GeckoChannelIdentifier - if err := json.Unmarshal([]byte(identifier), &channelIdent); err != nil { - return - } - - // Find which pool this is - var poolChain string - for _, pool := range geckoTerminalPools { - if pool.PoolID == channelIdent.PoolID { - poolChain = pool.Chain - break - } - } - - if poolChain == "" { - return - } - - // Calculate head lag - receiveTime := time.Now().UTC() - onChainTime := time.UnixMilli(swapData.Data.BlockTimestamp) - lagMs := receiveTime.Sub(onChainTime).Milliseconds() - lagSeconds := float64(lagMs) / 1000.0 - - // Record metrics with tx hash - RecordHeadLag("geckoterminal", poolChain, lagMs, lagSeconds, config.MonitorRegion, swapData.Data.TxHash) - - // Log occasionally (not every trade) - if lagMs > 10000 || time.Now().Second()%30 == 0 { - timestamp := receiveTime.Format("15:04:05") - txHash := swapData.Data.TxHash - if len(txHash) > 12 { - txHash = txHash[:10] + "..." - } - fmt.Printf("[HEAD-LAG][GECKO][%s][%s] Lag: %.2fs | Tx: %s\n", - timestamp, poolChain, lagSeconds, txHash) - } -} - -func subscribeToGeckoSwapChannel(conn *websocket.Conn, poolID, poolName string) { - identifier := GeckoChannelIdentifier{ - Channel: "SwapChannel", - PoolID: poolID, - } - - identifierJSON, _ := json.Marshal(identifier) - - subscribeMsg := GeckoActionCableMessage{ - Command: "subscribe", - Identifier: string(identifierJSON), - } - - if err := conn.WriteJSON(subscribeMsg); err != nil { - log.Printf("[HEAD-LAG][GECKO] Error subscribing to %s: %v", poolName, err) - return - } -} diff --git a/harnesses/metadata-coverage/cmd/script/head_lag_monitor.go b/harnesses/metadata-coverage/cmd/script/head_lag_monitor.go deleted file mode 100644 index 7f1dfef9..00000000 --- a/harnesses/metadata-coverage/cmd/script/head_lag_monitor.go +++ /dev/null @@ -1,673 +0,0 @@ -package main - -import ( - "encoding/json" - "fmt" - "log" - "strings" - "sync" - "time" - - "github.com/gorilla/websocket" -) - -// ============================================================================ -// Head Lag Monitor -// Measures indexation latency: time between on-chain event and WebSocket receipt -// ============================================================================ - -// Pool configurations for head lag monitoring -type HeadLagPool struct { - Name string // Human readable name - Blockchain string // For Mobula: "evm:1", "solana", etc. - NetworkID int // For Codex: 1, 1399811149, etc. - Address string // Pool address - ChainName string // Normalized chain name for metrics -} - -// Pools to monitor - high activity pools for accurate lag measurement -var headLagPools = []HeadLagPool{ - { - Name: "SOL/USDC Raydium", - Blockchain: "solana", - NetworkID: 1399811149, - Address: "7qbRF6YsyGuLUVs6Y1q64bdVrfe4ZcUUz1JRdoVNUJnm", - ChainName: "solana", - }, - { - Name: "WETH/USDC Base", - Blockchain: "evm:8453", - NetworkID: 8453, - Address: "0x4c36388be6f416a29c8d8eee81c771ce6be14b18", - ChainName: "base", - }, - { - Name: "WBNB/BUSD PancakeSwap", - Blockchain: "evm:56", - NetworkID: 56, - Address: "0x58f876857a02d6762e0101bb5c46a8c1ed44dc16", - ChainName: "bnb", - }, -} - -// ============================================================================ -// Mobula WebSocket Monitor -// ============================================================================ - -type MobulaTradeEvent struct { - Blockchain string `json:"blockchain"` - Date int64 `json:"date"` // On-chain timestamp (ms) - Timestamp int64 `json:"timestamp"` // When Mobula processed it (ms) - Hash string `json:"hash"` - Pair string `json:"pair"` - Type string `json:"type"` - TokenPrice float64 `json:"tokenPrice"` -} - -func runMobulaHeadLagMonitor(config *Config, stopChan <-chan struct{}, wg *sync.WaitGroup) { - defer wg.Done() - - if config.MobulaAPIKey == "" { - fmt.Println("[HEAD-LAG][MOBULA] API key not set, skipping") - return - } - - fmt.Println("[HEAD-LAG][MOBULA] Starting WebSocket monitor...") - - reconnectDelay := 5 * time.Second - maxReconnectDelay := 60 * time.Second - - for { - select { - case <-stopChan: - fmt.Println("[HEAD-LAG][MOBULA] Monitor stopped") - return - default: - err := connectAndMonitorMobula(config, stopChan) - if err != nil { - log.Printf("[HEAD-LAG][MOBULA] Connection error: %v. Reconnecting in %v...", err, reconnectDelay) - - select { - case <-stopChan: - return - case <-time.After(reconnectDelay): - reconnectDelay = reconnectDelay * 2 - if reconnectDelay > maxReconnectDelay { - reconnectDelay = maxReconnectDelay - } - } - } else { - // Reset delay on clean disconnect - reconnectDelay = 5 * time.Second - } - } - } -} - -func connectAndMonitorMobula(config *Config, stopChan <-chan struct{}) error { - // Use direct connection for Mobula (no proxy needed, avoids latency spikes) - dialer := &websocket.Dialer{} - conn, _, err := dialer.Dial(config.MobulaWSURL, nil) - if err != nil { - return fmt.Errorf("dial failed: %w", err) - } - defer conn.Close() - - // Build subscription items - var items []map[string]interface{} - for _, pool := range headLagPools { - items = append(items, map[string]interface{}{ - "blockchain": pool.Blockchain, - "address": pool.Address, - }) - } - - // Subscribe to fast-trade - subscribeMsg := map[string]interface{}{ - "type": "fast-trade", - "authorization": config.MobulaAPIKey, - "payload": map[string]interface{}{ - "assetMode": false, - "items": items, - }, - } - - if err := conn.WriteJSON(subscribeMsg); err != nil { - return fmt.Errorf("subscribe failed: %w", err) - } - - fmt.Printf("[HEAD-LAG][MOBULA] Subscribed to %d pools\n", len(items)) - - // Start ping goroutine - pingDone := make(chan struct{}) - go func() { - ticker := time.NewTicker(25 * time.Second) - defer ticker.Stop() - for { - select { - case <-pingDone: - return - case <-ticker.C: - if err := conn.WriteJSON(map[string]string{"event": "ping"}); err != nil { - return - } - } - } - }() - defer close(pingDone) - - // Read messages - for { - select { - case <-stopChan: - return nil - default: - conn.SetReadDeadline(time.Now().Add(60 * time.Second)) - _, message, err := conn.ReadMessage() - if err != nil { - return fmt.Errorf("read failed: %w", err) - } - - // Parse message - var trade MobulaTradeEvent - if err := json.Unmarshal(message, &trade); err != nil { - continue - } - - // Skip non-trade messages (pong, etc) - if trade.Hash == "" || trade.Date == 0 { - continue - } - - // Calculate head lag - receiveTime := time.Now().UTC() - onChainTime := time.UnixMilli(trade.Date) - mobulaProcessTime := time.UnixMilli(trade.Timestamp) - - // Total lag: on-chain → WebSocket receipt - totalLagMs := receiveTime.Sub(onChainTime).Milliseconds() - - // Drop WebSocket replays / clock-skew events: not real indexation latency - // (Mobula WS occasionally replays old trades on reconnect; those would otherwise fire alerts) - if totalLagMs < 0 || totalLagMs > 30000 { - continue - } - - // Mobula processing latency: on-chain → Mobula processed - mobulaLagMs := mobulaProcessTime.Sub(onChainTime).Milliseconds() - - // Network latency: Mobula processed → WebSocket receipt - networkLagMs := receiveTime.Sub(mobulaProcessTime).Milliseconds() - - lagSeconds := float64(totalLagMs) / 1000.0 - - // Get chain name from pool config - chainName := getChainNameFromBlockchain(trade.Blockchain) - - // Record basic metric - RecordHeadLag("mobula", chainName, totalLagMs, lagSeconds, config.MonitorRegion, trade.Hash) - - // Track latest tx per pool so alert annotations can link to it - RecordMobulaLastTx(chainName, config.MonitorRegion, trade.Pair, trade.Hash) - - // Record detailed metric with breakdown - RecordMobulaHeadLagDetailed( - chainName, - config.MonitorRegion, - trade.Pair, - trade.Hash, - totalLagMs, - mobulaLagMs, - networkLagMs, - onChainTime.Format("2006-01-02T15:04:05Z"), - mobulaProcessTime.Format("2006-01-02T15:04:05Z"), - receiveTime.Format("2006-01-02T15:04:05Z"), - ) - - // Enhanced logging for spikes - if totalLagMs > 3000 { - timestamp := receiveTime.Format("15:04:05") - fmt.Printf("[HEAD-LAG][MOBULA][%s][%s] 🚨 SPIKE DETECTED!\n", timestamp, chainName) - fmt.Printf(" Total Lag: %.2fs (%.0fms)\n", lagSeconds, float64(totalLagMs)) - fmt.Printf(" ├─ Mobula Processing: %.0fms (on-chain → Mobula)\n", float64(mobulaLagMs)) - fmt.Printf(" └─ Network Latency: %.0fms (Mobula → WebSocket)\n", float64(networkLagMs)) - fmt.Printf(" On-chain: %s | Mobula: %s | Received: %s\n", - onChainTime.Format("15:04:05.000"), - mobulaProcessTime.Format("15:04:05.000"), - receiveTime.Format("15:04:05.000")) - fmt.Printf(" Tx: %s\n", trade.Hash) - } else if time.Now().Second()%30 == 0 { - // Log normal latency occasionally - timestamp := receiveTime.Format("15:04:05") - fmt.Printf("[HEAD-LAG][MOBULA][%s][%s] Lag: %.2fs | Tx: %s\n", - timestamp, chainName, lagSeconds, trade.Hash[:12]+"...") - } - } - } -} - -func getChainNameFromBlockchain(blockchain string) string { - switch blockchain { - case "Ethereum", "evm:1": - return "ethereum" - case "Solana", "solana": - return "solana" - case "Base", "evm:8453": - return "base" - case "BNB Smart Chain (BEP20)", "BSC", "evm:56": - return "bnb" - default: - return blockchain - } -} - -// ============================================================================ -// Codex WebSocket Monitor (using Defined.fi session auth) -// ============================================================================ - -type CodexWSMessage struct { - Type string `json:"type"` - ID string `json:"id,omitempty"` - Payload map[string]interface{} `json:"payload,omitempty"` -} - -type CodexEventData struct { - Data struct { - OnEventsCreated *struct { - Address string `json:"address"` - NetworkID int `json:"networkId"` - Events []struct { - BlockNumber int64 `json:"blockNumber"` - Timestamp int64 `json:"timestamp"` - TransactionHash string `json:"transactionHash"` - EventType string `json:"eventType"` - } `json:"events"` - } `json:"onEventsCreated,omitempty"` - OnUnconfirmedEventsCreated *struct { - Address string `json:"address"` - NetworkID int `json:"networkId"` - Events []struct { - BlockNumber int64 `json:"blockNumber"` - Timestamp int64 `json:"timestamp"` - TransactionHash string `json:"transactionHash"` - EventType string `json:"eventType"` - } `json:"events"` - } `json:"onUnconfirmedEventsCreated,omitempty"` - } `json:"data"` -} - -func runCodexHeadLagMonitor(config *Config, stopChan <-chan struct{}, wg *sync.WaitGroup) { - defer wg.Done() - - fmt.Println("[HEAD-LAG][CODEX] Starting WebSocket monitor (via Defined.fi auth)...") - - // Backoff strategy: - // - 10s base delay (proxy IP rotation window) - // - Doubles on consecutive net errors, capped at 60s (NOT 5min — too long - // for the rotating-proxy case; we want to keep probing for fresh IPs) - // - Every 10 consecutive failures, force-reset to 10s AND invalidate the JWT - // cache (covers silent token expiry that surfaces as net errors) - const baseDelay = 10 * time.Second - const maxReconnectDelay = 60 * time.Second - const forceResetEvery = 10 - - reconnectDelay := baseDelay - consecutiveFailures := 0 - attemptNum := 0 - - for { - select { - case <-stopChan: - fmt.Println("[HEAD-LAG][CODEX] Monitor stopped") - return - default: - attemptNum++ - log.Printf("[HEAD-LAG][CODEX] 🔄 Connection attempt #%d (proxy rotation enabled)", attemptNum) - - err := connectAndMonitorCodex(config, stopChan) - if err != nil { - consecutiveFailures++ - log.Printf("[HEAD-LAG][CODEX] ❌ Connection attempt #%d failed (%d consecutive): %v", attemptNum, consecutiveFailures, err) - - if strings.Contains(err.Error(), "rate limited (429)") || strings.Contains(err.Error(), "too many token requests") { - log.Printf("[HEAD-LAG][CODEX] ⚠️ Rate limited on JWT — invalidating cache, retrying in 30s") - InvalidateTokenCache() - reconnectDelay = 30 * time.Second - } else if strings.Contains(err.Error(), "authentication") || strings.Contains(err.Error(), "401") || strings.Contains(err.Error(), "4401") || strings.Contains(err.Error(), "4403") { - log.Printf("[HEAD-LAG][CODEX] 🔑 Auth error — invalidating JWT cache, retrying in 30s") - InvalidateTokenCache() - reconnectDelay = 30 * time.Second - } else { - // Net/IO error: exponential backoff, capped at 60s. - reconnectDelay *= 2 - if reconnectDelay < baseDelay { - reconnectDelay = baseDelay - } - if reconnectDelay > maxReconnectDelay { - reconnectDelay = maxReconnectDelay - } - } - - // Periodic force-reset: stale JWT silently rejected can manifest - // as net errors (server-closed connection). Every 10 failures, - // invalidate the token AND drop back to base delay so we probe - // fast on a fresh state. - if consecutiveFailures%forceResetEvery == 0 { - log.Printf("[HEAD-LAG][CODEX] 🔁 %d consecutive failures — force-reset (invalidate JWT + base delay)", consecutiveFailures) - InvalidateTokenCache() - reconnectDelay = baseDelay - } - - log.Printf("[HEAD-LAG][CODEX] ⏳ Reconnecting in %v... (next attempt: #%d)", reconnectDelay, attemptNum+1) - select { - case <-stopChan: - return - case <-time.After(reconnectDelay): - } - } else { - log.Printf("[HEAD-LAG][CODEX] ✅ Connection closed cleanly after attempt #%d", attemptNum) - reconnectDelay = baseDelay - consecutiveFailures = 0 - attemptNum = 0 - } - } - } -} - -func connectAndMonitorCodex(config *Config, stopChan <-chan struct{}) error { - log.Printf("[HEAD-LAG][CODEX] Step 1/4: Generating JWT token...") - jwtToken, err := GetDefinedJWTToken(config.DefinedSessionCookie) - if err != nil { - return fmt.Errorf("failed to get JWT token: %w", err) - } - - log.Printf("[HEAD-LAG][CODEX] Step 2/4: Creating proxy dialer...") - dialer := getProxyDialerWithSubprotocols([]string{"graphql-transport-ws"}) - - log.Printf("[HEAD-LAG][CODEX] Step 3/4: Connecting to wss://graph.codex.io/graphql...") - conn, resp, err := dialer.Dial("wss://graph.codex.io/graphql", nil) - if err != nil { - if resp != nil { - return fmt.Errorf("dial failed (HTTP %d): %w", resp.StatusCode, err) - } - return fmt.Errorf("dial failed: %w", err) - } - defer conn.Close() - - // Verify proxy IP rotation (for debugging) - if outboundIP, err := getOutboundIP(); err == nil { - log.Printf("[HEAD-LAG][CODEX] Step 3/4: ✅ WebSocket connection established via proxy IP: %s", outboundIP) - } else { - log.Printf("[HEAD-LAG][CODEX] Step 3/4: ✅ WebSocket connection established (IP check failed: %v)", err) - } - - // Connection init with JWT Bearer token - log.Printf("[HEAD-LAG][CODEX] Step 4/4: Sending connection_init with JWT...") - initMsg := map[string]interface{}{ - "type": "connection_init", - "payload": map[string]interface{}{ - "Authorization": fmt.Sprintf("Bearer %s", jwtToken), - }, - } - if err := conn.WriteJSON(initMsg); err != nil { - return fmt.Errorf("init failed: %w", err) - } - - // Wait for ack - conn.SetReadDeadline(time.Now().Add(10 * time.Second)) - _, msg, err := conn.ReadMessage() - if err != nil { - return fmt.Errorf("ack read failed: %w", err) - } - - var ackMsg CodexWSMessage - if err := json.Unmarshal(msg, &ackMsg); err != nil || ackMsg.Type != "connection_ack" { - return fmt.Errorf("unexpected ack: %s", string(msg)) - } - log.Printf("[HEAD-LAG][CODEX] Step 4/4: ✅ Received connection_ack") - - // Subscribe to each pool (use onUnconfirmedEventsCreated for Solana, onEventsCreated for others) - log.Printf("[HEAD-LAG][CODEX] Subscribing to %d pools...", len(headLagPools)) - for i, pool := range headLagPools { - subID := fmt.Sprintf("headlag_%d", i) - - var subMsg map[string]interface{} - - // Solana: use onUnconfirmedEventsCreated (pre-finalized, lowest latency) - if pool.ChainName == "solana" { - poolID := fmt.Sprintf("%s:%d", pool.Address, pool.NetworkID) - subMsg = map[string]interface{}{ - "type": "subscribe", - "id": subID, - "payload": map[string]interface{}{ - "query": `subscription OnPoolEvents($id: String!) { - onUnconfirmedEventsCreated(id: $id, quoteToken: token0) { - address - networkId - events { - blockNumber - timestamp - transactionHash - eventType - } - } - }`, - "variables": map[string]interface{}{ - "id": poolID, - }, - }, - } - } else { - // EVM chains: use onEventsCreated (confirmed events) - subMsg = map[string]interface{}{ - "type": "subscribe", - "id": subID, - "payload": map[string]interface{}{ - "query": `subscription OnPoolEvents($address: String!, $networkId: Int!) { - onEventsCreated(address: $address, networkId: $networkId) { - address - networkId - events { - blockNumber - timestamp - transactionHash - eventType - } - } - }`, - "variables": map[string]interface{}{ - "address": pool.Address, - "networkId": pool.NetworkID, - }, - }, - } - } - - if err := conn.WriteJSON(subMsg); err != nil { - return fmt.Errorf("subscribe to %s failed: %w", pool.Name, err) - } - - time.Sleep(100 * time.Millisecond) // Small delay between subscriptions - } - - log.Printf("[HEAD-LAG][CODEX] ✅ Subscribed to %d pools", len(headLagPools)) - log.Printf("[HEAD-LAG][CODEX] 🎉 Connection fully established! Waiting for events...") - - // graphql-transport-ws keepalive: send {"type":"ping"} every 20s. - // Without this, the rotating proxy (or Codex itself) silently kills idle - // TCP after ~3min, manifesting as "i/o timeout" or "1006 unexpected EOF". - // Writes to the websocket conn are serialized via writeMu since we now have - // two goroutines writing (this ping loop + the subscribe path on next reconnect). - var writeMu sync.Mutex - pingDone := make(chan struct{}) - go func() { - t := time.NewTicker(20 * time.Second) - defer t.Stop() - for { - select { - case <-pingDone: - return - case <-t.C: - writeMu.Lock() - err := conn.WriteJSON(map[string]string{"type": "ping"}) - writeMu.Unlock() - if err != nil { - return // read loop will see the failure too and exit cleanly - } - } - } - }() - defer close(pingDone) - - // Read messages - for { - select { - case <-stopChan: - return nil - default: - // 90s deadline: longer than ping interval so a single missed pong doesn't - // kill us, but short enough to fail fast if the server actually went away. - conn.SetReadDeadline(time.Now().Add(90 * time.Second)) - _, message, err := conn.ReadMessage() - if err != nil { - return fmt.Errorf("read failed: %w", err) - } - - // Parse message - var wsMsg CodexWSMessage - if err := json.Unmarshal(message, &wsMsg); err != nil { - continue - } - - // graphql-transport-ws keepalive: server may push ping; reply pong. - // Server may also push pong in response to our ping — silently consume. - if wsMsg.Type == "ping" { - writeMu.Lock() - _ = conn.WriteJSON(map[string]string{"type": "pong"}) - writeMu.Unlock() - continue - } - if wsMsg.Type == "pong" { - continue - } - - // Skip non-data messages - if wsMsg.Type != "next" || wsMsg.Payload == nil { - continue - } - - // Parse event data - payloadBytes, _ := json.Marshal(wsMsg.Payload) - var eventData CodexEventData - if err := json.Unmarshal(payloadBytes, &eventData); err != nil { - continue - } - - // Handle both onEventsCreated and onUnconfirmedEventsCreated - var events []struct { - BlockNumber int64 `json:"blockNumber"` - Timestamp int64 `json:"timestamp"` - TransactionHash string `json:"transactionHash"` - EventType string `json:"eventType"` - } - var networkID int - - if eventData.Data.OnEventsCreated != nil { - events = eventData.Data.OnEventsCreated.Events - networkID = eventData.Data.OnEventsCreated.NetworkID - } else if eventData.Data.OnUnconfirmedEventsCreated != nil { - events = eventData.Data.OnUnconfirmedEventsCreated.Events - networkID = eventData.Data.OnUnconfirmedEventsCreated.NetworkID - } - - if len(events) == 0 { - continue - } - - for _, event := range events { - if event.EventType != "Swap" || event.TransactionHash == "" { - continue - } - - // Calculate head lag - receiveTime := time.Now().UTC() - onChainTime := time.Unix(event.Timestamp, 0) - lagMs := receiveTime.Sub(onChainTime).Milliseconds() - lagSeconds := float64(lagMs) / 1000.0 - - // Get chain name - chainName := getChainNameFromNetworkID(networkID) - - // Record metrics with tx hash - RecordHeadLag("codex", chainName, lagMs, lagSeconds, config.MonitorRegion, event.TransactionHash) - RecordCodexBlockNumber(chainName, event.BlockNumber, config.MonitorRegion) - - // Enhanced logging for spikes - if lagMs > 3000 { - timestamp := receiveTime.Format("15:04:05") - fmt.Printf("[HEAD-LAG][CODEX][%s][%s] 🚨 SPIKE DETECTED!\n", timestamp, chainName) - fmt.Printf(" Total Lag: %.2fs (%.0fms)\n", lagSeconds, float64(lagMs)) - fmt.Printf(" On-chain: %s | Received: %s\n", - onChainTime.Format("15:04:05.000"), - receiveTime.Format("15:04:05.000")) - fmt.Printf(" Block: %d | Tx: %s\n", event.BlockNumber, event.TransactionHash) - } else if time.Now().Second()%30 == 0 { - // Log normal latency occasionally - timestamp := receiveTime.Format("15:04:05") - fmt.Printf("[HEAD-LAG][CODEX][%s][%s] Lag: %.2fs | Block: %d | Tx: %s\n", - timestamp, chainName, lagSeconds, event.BlockNumber, event.TransactionHash[:12]+"...") - } - } - } - } -} - -func getChainNameFromNetworkID(networkID int) string { - switch networkID { - case 1: - return "ethereum" - case 1399811149: - return "solana" - case 8453: - return "base" - case 56: - return "bnb" - default: - return fmt.Sprintf("network_%d", networkID) - } -} - -// ============================================================================ -// Main Head Lag Monitor -// ============================================================================ - -func runHeadLagMonitor(config *Config, stopChan <-chan struct{}) { - fmt.Println() - fmt.Println("╔══════════════════════════════════════════════════════════════╗") - fmt.Println("║ HEAD LAG MONITOR (WebSocket-based) ║") - fmt.Println("╠══════════════════════════════════════════════════════════════╣") - fmt.Println("║ Measures: Time between on-chain event and WebSocket receipt ║") - fmt.Println("║ Providers: Mobula + Pulse + Codex + GeckoTerminal ║") - fmt.Printf("║ Pools: %d high-activity pools across 3 chains ║\n", len(headLagPools)) - fmt.Printf("║ Region: %s ║\n", config.MonitorRegion) - fmt.Println("╚══════════════════════════════════════════════════════════════╝") - fmt.Println() - - var wg sync.WaitGroup - - // Start Mobula fast-trade monitor - wg.Add(1) - go runMobulaHeadLagMonitor(config, stopChan, &wg) - - // Start Codex monitor - wg.Add(1) - go runCodexHeadLagMonitor(config, stopChan, &wg) - - // Start GeckoTerminal monitor - wg.Add(1) - go runGeckoTerminalHeadLagMonitor(config, stopChan, &wg) - - // Wait for all to finish - wg.Wait() - fmt.Println("[HEAD-LAG] All monitors stopped") -} diff --git a/harnesses/metadata-coverage/cmd/script/log_buffer.go b/harnesses/metadata-coverage/cmd/script/log_buffer.go deleted file mode 100644 index 0f7455e0..00000000 --- a/harnesses/metadata-coverage/cmd/script/log_buffer.go +++ /dev/null @@ -1,102 +0,0 @@ -package main - -import ( - "bufio" - "fmt" - "io" - "net/http" - "os" - "strconv" - "sync" - "time" -) - -// logBuffer keeps the last N log lines in memory for debug fetching via /logs. -// Captures BOTH log.* and fmt.Print* output (stdout is dup'd via a pipe). -type logBuffer struct { - mu sync.Mutex - lines []string - max int -} - -const logBufferMax = 5000 - -var globalLogBuffer = &logBuffer{max: logBufferMax} - -func (b *logBuffer) push(line string) { - entry := time.Now().UTC().Format("2006-01-02T15:04:05.000Z") + " " + line - b.mu.Lock() - if len(b.lines) >= b.max { - b.lines = append(b.lines[1:], entry) - } else { - b.lines = append(b.lines, entry) - } - b.mu.Unlock() -} - -func (b *logBuffer) Snapshot(tail int) []string { - b.mu.Lock() - defer b.mu.Unlock() - if tail <= 0 || tail >= len(b.lines) { - out := make([]string, len(b.lines)) - copy(out, b.lines) - return out - } - start := len(b.lines) - tail - out := make([]string, tail) - copy(out, b.lines[start:]) - return out -} - -// installLogCapture replaces os.Stdout with the write-end of a pipe, then -// spawns a goroutine that fan-outs every line to the real stdout AND the -// in-memory ring buffer. This catches fmt.Println/Printf as well as log.Printf. -// -// Call exactly once, very early in main(). -func installLogCapture() { - originalStdout := os.Stdout - r, w, err := os.Pipe() - if err != nil { - // Fallback: don't intercept. /logs will be empty but everything else still works. - fmt.Fprintf(originalStdout, "[log_buffer] failed to create pipe: %v (logs endpoint will be empty)\n", err) - return - } - os.Stdout = w - - go func() { - scanner := bufio.NewScanner(r) - // Allow long lines (default is 64KB, bump to 1MB for safety) - buf := make([]byte, 0, 1024*1024) - scanner.Buffer(buf, 1024*1024) - for scanner.Scan() { - line := scanner.Text() - fmt.Fprintln(originalStdout, line) - globalLogBuffer.push(line) - } - // Pipe closed (process shutdown) — drain anything left - _, _ = io.Copy(originalStdout, r) - }() -} - -// setupLogsEndpoint exposes GET /logs?tail=N (default 500, max logBufferMax). -// If LOGS_TOKEN env var is set, requires header `X-Logs-Token` to match. -// Otherwise the endpoint is open — only safe for Railway-internal access. -func setupLogsEndpoint(mux *http.ServeMux) { - expectedToken := os.Getenv("LOGS_TOKEN") - mux.HandleFunc("/logs", func(w http.ResponseWriter, r *http.Request) { - if expectedToken != "" && r.Header.Get("X-Logs-Token") != expectedToken { - http.Error(w, "forbidden", http.StatusForbidden) - return - } - tail := 500 - if t := r.URL.Query().Get("tail"); t != "" { - if n, err := strconv.Atoi(t); err == nil && n > 0 { - tail = n - } - } - w.Header().Set("Content-Type", "text/plain; charset=utf-8") - for _, l := range globalLogBuffer.Snapshot(tail) { - fmt.Fprintln(w, l) - } - }) -} diff --git a/harnesses/metadata-coverage/cmd/script/main.go b/harnesses/metadata-coverage/cmd/script/main.go deleted file mode 100644 index a7eb31e7..00000000 --- a/harnesses/metadata-coverage/cmd/script/main.go +++ /dev/null @@ -1,73 +0,0 @@ -package main - -import ( - "fmt" - "os" - "os/signal" - "sync" - "syscall" -) - -func main() { - installLogCapture() // must be first — captures all subsequent stdout into ring buffer for /logs - fmt.Println("=== Aggregator Indexation Lag Monitor ===") - fmt.Println("Measuring real-time indexation lag (head lag) for blockchain data APIs") - fmt.Println("Press Ctrl+C to stop") - fmt.Println() - - config, err := loadEnv() - if err != nil { - fmt.Printf("Error: %v\n", err) - os.Exit(1) - } - - // Use session cookie from environment (scraping requires GUI, doesn't work on Railway) - if config.DefinedSessionCookie == "" { - fmt.Println("Warning: DEFINED_SESSION_COOKIE not set in environment") - fmt.Println("Codex REST and WebSocket monitors will not work") - } else { - fmt.Printf("Using DEFINED_SESSION_COOKIE from environment (length: %d)\n", len(config.DefinedSessionCookie)) - } - - fmt.Println("Metrics will be exposed on :2112/metrics for Prometheus") - fmt.Println() - - sigChan := make(chan os.Signal, 1) - signal.Notify(sigChan, os.Interrupt, syscall.SIGTERM) - - var wg sync.WaitGroup - stopChan := make(chan struct{}) - - wg.Add(1) - go func() { - defer wg.Done() - fmt.Println("Starting Prometheus metrics server on :2112") - if err := StartMetricsServer(":2112"); err != nil { - fmt.Printf("Metrics server error: %v\n", err) - } - }() - - // Mobula Pulse V2 feeder — discovers fresh tokens and pushes them - // onto the metadata-coverage queue. - wg.Add(1) - go func() { - defer wg.Done() - runMobulaPulseMonitor(config, stopChan) - }() - - // Metadata coverage monitor — the only data producer. Consumes the - // queue, queries each aggregator's metadata endpoint, records field - // coverage (logo, description, twitter, website). - wg.Add(1) - go func() { - defer wg.Done() - runMetadataCoverageMonitor(config, stopChan) - }() - - <-sigChan - fmt.Println("\n\nShutting down monitors...") - close(stopChan) - - wg.Wait() - fmt.Println("All monitors stopped") -} diff --git a/harnesses/metadata-coverage/cmd/script/metadata_coverage_monitor.go b/harnesses/metadata-coverage/cmd/script/metadata_coverage_monitor.go deleted file mode 100644 index 0438ecc3..00000000 --- a/harnesses/metadata-coverage/cmd/script/metadata_coverage_monitor.go +++ /dev/null @@ -1,735 +0,0 @@ -package main - -import ( - "bytes" - "encoding/json" - "fmt" - "io" - "net/http" - "net/url" - "sync" - "time" -) - -// ============================================================================ -// Metadata Coverage Monitor -// Measures metadata and logo coverage across providers (Mobula, Codex) -// ============================================================================ - -const ( - mobulaTokenDetailsURL = "https://api.mobula.io/api/2/token/details" - codexGraphQLURL = "https://graph.codex.io/graphql" - jupiterTokenPageURL = "https://jup.ag/tokens/" -) - -// TokenToCheck represents a token discovered via Pulse that needs metadata checking -type TokenToCheck struct { - Address string - ChainID string // e.g., "solana", "evm:1", "evm:8453" - Symbol string - Name string - DetectedAt time.Time -} - -// MetadataFields represents the fields we check for coverage -type MetadataFields struct { - HasLogo bool - HasName bool - HasSymbol bool - HasDescription bool - HasTwitter bool - HasWebsite bool - HasTelegram bool - LogoURL string - ResponseTimeMs float64 - Error string -} - -// ProviderCoverage holds coverage stats for a single provider -type ProviderCoverage struct { - Provider string - TotalChecks int - LogoCount int - NameCount int - SymbolCount int - DescCount int - TwitterCount int - WebsiteCount int - TelegramCount int - ErrorCount int - TotalLatencyMs float64 -} - -// MetadataCoverageStats holds overall stats -type MetadataCoverageStats struct { - mu sync.Mutex - Mobula ProviderCoverage - Codex ProviderCoverage - Jupiter ProviderCoverage - LastPrint time.Time -} - -var ( - coverageStats = &MetadataCoverageStats{ - Mobula: ProviderCoverage{Provider: "mobula"}, - Codex: ProviderCoverage{Provider: "codex"}, - Jupiter: ProviderCoverage{Provider: "jupiter"}, - } - tokenQueue = make(chan TokenToCheck, 500) - metadataClient = &http.Client{Timeout: 10 * time.Second} -) - -// ============================================================================ -// Mobula API - Token Details -// ============================================================================ - -type MobulaTokenDetailsResponse struct { - Data MobulaTokenData `json:"data"` -} - -type MobulaTokenData struct { - Address string `json:"address"` - Name string `json:"name"` - Symbol string `json:"symbol"` - Logo string `json:"logo"` - Description string `json:"description"` - Socials MobulaSocials `json:"socials"` -} - -type MobulaSocials struct { - Twitter string `json:"twitter"` - Website string `json:"website"` - Telegram string `json:"telegram"` -} - -func checkMobulaMetadata(token TokenToCheck, apiKey string) MetadataFields { - result := MetadataFields{} - - // Build URL with query params - params := url.Values{} - params.Add("address", token.Address) - - // Convert chainID to Mobula format - blockchain := token.ChainID - if blockchain == "solana:solana" { - blockchain = "solana" - } - params.Add("blockchain", blockchain) - - fullURL := fmt.Sprintf("%s?%s", mobulaTokenDetailsURL, params.Encode()) - - req, err := http.NewRequest("GET", fullURL, nil) - if err != nil { - result.Error = fmt.Sprintf("request_create_error: %v", err) - return result - } - - req.Header.Set("Accept", "application/json") - if apiKey != "" { - req.Header.Set("Authorization", apiKey) - } - - startTime := time.Now() - resp, err := metadataClient.Do(req) - result.ResponseTimeMs = float64(time.Since(startTime).Milliseconds()) - - if err != nil { - result.Error = fmt.Sprintf("request_error: %v", err) - return result - } - defer resp.Body.Close() - - if resp.StatusCode != 200 { - result.Error = fmt.Sprintf("status_%d", resp.StatusCode) - return result - } - - body, err := io.ReadAll(resp.Body) - if err != nil { - result.Error = fmt.Sprintf("read_error: %v", err) - return result - } - - var response MobulaTokenDetailsResponse - if err := json.Unmarshal(body, &response); err != nil { - result.Error = fmt.Sprintf("parse_error: %v", err) - return result - } - - data := response.Data - - // Check each field - result.HasName = data.Name != "" - result.HasSymbol = data.Symbol != "" - result.HasLogo = data.Logo != "" - result.LogoURL = data.Logo - result.HasDescription = data.Description != "" - result.HasTwitter = data.Socials.Twitter != "" - result.HasWebsite = data.Socials.Website != "" - result.HasTelegram = data.Socials.Telegram != "" - - return result -} - -// ============================================================================ -// Codex API - GraphQL Token Query -// ============================================================================ - -// Note: CodexGraphQLRequest is defined in codex_rest_monitor.go - -// CodexTokenResponse represents the response from token query -// Returns EnhancedToken with socialLinks and info -// https://docs.codex.io/api-reference/queries/token -type CodexTokenResponse struct { - Data struct { - Token CodexEnhancedToken `json:"token"` - } `json:"data"` - Errors []struct { - Message string `json:"message"` - } `json:"errors"` -} - -// CodexEnhancedToken matches the EnhancedToken type from Codex API -type CodexEnhancedToken struct { - Address string `json:"address"` - Name string `json:"name"` - Symbol string `json:"symbol"` - Decimals int `json:"decimals"` - NetworkID int `json:"networkId"` - Info *CodexTokenInfo `json:"info"` - SocialLinks *CodexSocialLinks `json:"socialLinks"` -} - -// CodexTokenInfo contains metadata about the token -type CodexTokenInfo struct { - ImageThumbUrl string `json:"imageThumbUrl"` - ImageSmallUrl string `json:"imageSmallUrl"` - ImageLargeUrl string `json:"imageLargeUrl"` - Description string `json:"description"` - CirculatingSupply string `json:"circulatingSupply"` - TotalSupply string `json:"totalSupply"` -} - -// CodexSocialLinks contains social media links for the token -type CodexSocialLinks struct { - Twitter string `json:"twitter"` - Website string `json:"website"` - Telegram string `json:"telegram"` - Discord string `json:"discord"` - Github string `json:"github"` -} - -func getCodexNetworkID(chainID string) int { - switch chainID { - case "solana", "solana:solana": - return 1399811149 - case "evm:1": - return 1 - case "evm:8453": - return 8453 - case "evm:56": - return 56 - case "evm:42161": - return 42161 - default: - return 0 - } -} - -func checkCodexMetadata(token TokenToCheck, sessionCookie string) MetadataFields { - result := MetadataFields{} - - networkID := getCodexNetworkID(token.ChainID) - if networkID == 0 { - result.Error = "unsupported_chain" - return result - } - - // Codex GraphQL no longer accepts the raw Defined session cookie as Bearer - // (returns 401 UNAUTHENTICATED). Mint a JWT from the cookie via defined.fi/api - // and use that as Bearer — same path as the head_lag WS monitor. - apiKey, err := GetDefinedJWTToken(sessionCookie) - if err != nil { - result.Error = fmt.Sprintf("jwt_mint_error: %v", err) - return result - } - - // Use token query which returns EnhancedToken with socialLinks and info - // https://docs.codex.io/api-reference/queries/token - query := `query GetToken($address: String!, $networkId: Int!) { - token(input: { address: $address, networkId: $networkId }) { - address - name - symbol - decimals - networkId - info { - imageThumbUrl - imageSmallUrl - imageLargeUrl - description - circulatingSupply - totalSupply - } - socialLinks { - twitter - website - telegram - discord - github - } - } - }` - - reqBody := CodexGraphQLRequest{ - Query: query, - Variables: map[string]interface{}{ - "address": token.Address, - "networkId": networkID, - }, - } - - jsonBody, err := json.Marshal(reqBody) - if err != nil { - result.Error = fmt.Sprintf("marshal_error: %v", err) - return result - } - - req, err := http.NewRequest("POST", codexGraphQLURL, bytes.NewBuffer(jsonBody)) - if err != nil { - result.Error = fmt.Sprintf("request_create_error: %v", err) - return result - } - - req.Header.Set("Content-Type", "application/json") - req.Header.Set("Accept", "application/json") - if apiKey != "" { - req.Header.Set("Authorization", fmt.Sprintf("Bearer %s", apiKey)) - } - - startTime := time.Now() - resp, err := metadataClient.Do(req) - result.ResponseTimeMs = float64(time.Since(startTime).Milliseconds()) - - if err != nil { - result.Error = fmt.Sprintf("request_error: %v", err) - return result - } - defer resp.Body.Close() - - if resp.StatusCode != 200 { - result.Error = fmt.Sprintf("status_%d", resp.StatusCode) - return result - } - - body, err := io.ReadAll(resp.Body) - if err != nil { - result.Error = fmt.Sprintf("read_error: %v", err) - return result - } - - var response CodexTokenResponse - if err := json.Unmarshal(body, &response); err != nil { - result.Error = fmt.Sprintf("parse_error: %v", err) - return result - } - - if len(response.Errors) > 0 { - result.Error = fmt.Sprintf("graphql_error: %s", response.Errors[0].Message) - return result - } - - data := response.Data.Token - - // Check if token was found - if data.Address == "" { - result.Error = "token_not_found" - return result - } - - // Check each field based on EnhancedToken - // https://docs.codex.io/api-reference/queries/token - result.HasName = data.Name != "" - result.HasSymbol = data.Symbol != "" - - // Check logo from info - if data.Info != nil { - result.HasLogo = data.Info.ImageThumbUrl != "" || data.Info.ImageSmallUrl != "" || data.Info.ImageLargeUrl != "" - if data.Info.ImageLargeUrl != "" { - result.LogoURL = data.Info.ImageLargeUrl - } else if data.Info.ImageSmallUrl != "" { - result.LogoURL = data.Info.ImageSmallUrl - } else { - result.LogoURL = data.Info.ImageThumbUrl - } - result.HasDescription = data.Info.Description != "" - } - - // Check social links - if data.SocialLinks != nil { - result.HasTwitter = data.SocialLinks.Twitter != "" - result.HasWebsite = data.SocialLinks.Website != "" - result.HasTelegram = data.SocialLinks.Telegram != "" - } - - return result -} - -// ============================================================================ -// Jupiter - Scraping from frontend (Solana only) -// ============================================================================ - -// JupiterNextData represents the __NEXT_DATA__ JSON structure -type JupiterNextData struct { - Props struct { - PageProps struct { - DehydratedState struct { - Queries []struct { - State struct { - Data JupiterTokenData `json:"data"` - } `json:"state"` - } `json:"queries"` - } `json:"dehydratedState"` - } `json:"pageProps"` - } `json:"props"` -} - -// JupiterTokenData represents token data from Jupiter -type JupiterTokenData struct { - ID string `json:"id"` - Name string `json:"name"` - Symbol string `json:"symbol"` - Icon string `json:"icon"` - Decimals int `json:"decimals"` -} - -func checkJupiterMetadata(token TokenToCheck) MetadataFields { - result := MetadataFields{} - - // Jupiter only supports Solana - if token.ChainID != "solana" && token.ChainID != "solana:solana" { - result.Error = "unsupported_chain" - return result - } - - // Scrape the token page - pageURL := jupiterTokenPageURL + token.Address - - req, err := http.NewRequest("GET", pageURL, nil) - if err != nil { - result.Error = fmt.Sprintf("request_create_error: %v", err) - return result - } - - req.Header.Set("User-Agent", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36") - req.Header.Set("Accept", "text/html,application/xhtml+xml") - - startTime := time.Now() - resp, err := metadataClient.Do(req) - result.ResponseTimeMs = float64(time.Since(startTime).Milliseconds()) - - if err != nil { - result.Error = fmt.Sprintf("request_error: %v", err) - return result - } - defer resp.Body.Close() - - if resp.StatusCode != 200 { - result.Error = fmt.Sprintf("status_%d", resp.StatusCode) - return result - } - - body, err := io.ReadAll(resp.Body) - if err != nil { - result.Error = fmt.Sprintf("read_error: %v", err) - return result - } - - // Extract __NEXT_DATA__ JSON from HTML - htmlContent := string(body) - startMarker := `` - - startIdx := -1 - for i := 0; i < len(htmlContent)-len(startMarker); i++ { - if htmlContent[i:i+len(startMarker)] == startMarker { - startIdx = i + len(startMarker) - break - } - } - - if startIdx == -1 { - result.Error = "next_data_not_found" - return result - } - - endIdx := -1 - for i := startIdx; i < len(htmlContent)-len(endMarker); i++ { - if htmlContent[i:i+len(endMarker)] == endMarker { - endIdx = i - break - } - } - - if endIdx == -1 { - result.Error = "next_data_end_not_found" - return result - } - - jsonData := htmlContent[startIdx:endIdx] - - var nextData JupiterNextData - if err := json.Unmarshal([]byte(jsonData), &nextData); err != nil { - result.Error = fmt.Sprintf("parse_error: %v", err) - return result - } - - // Find token data in queries - var tokenData JupiterTokenData - for _, query := range nextData.Props.PageProps.DehydratedState.Queries { - if query.State.Data.ID == token.Address { - tokenData = query.State.Data - break - } - } - - if tokenData.ID == "" { - result.Error = "token_not_found" - return result - } - - // Check fields - Jupiter only has basic on-chain data - result.HasName = tokenData.Name != "" - result.HasSymbol = tokenData.Symbol != "" - result.HasLogo = tokenData.Icon != "" - result.LogoURL = tokenData.Icon - // Jupiter doesn't have description or socials - result.HasDescription = false - result.HasTwitter = false - result.HasWebsite = false - result.HasTelegram = false - - return result -} - -// ============================================================================ -// Stats and Reporting -// ============================================================================ - -func updateStats(provider string, fields MetadataFields) { - coverageStats.mu.Lock() - defer coverageStats.mu.Unlock() - - var stats *ProviderCoverage - switch provider { - case "mobula": - stats = &coverageStats.Mobula - case "codex": - stats = &coverageStats.Codex - case "jupiter": - stats = &coverageStats.Jupiter - default: - return - } - - stats.TotalChecks++ - stats.TotalLatencyMs += fields.ResponseTimeMs - - if fields.Error != "" { - stats.ErrorCount++ - return - } - - if fields.HasLogo { - stats.LogoCount++ - } - if fields.HasName { - stats.NameCount++ - } - if fields.HasSymbol { - stats.SymbolCount++ - } - if fields.HasDescription { - stats.DescCount++ - } - if fields.HasTwitter { - stats.TwitterCount++ - } - if fields.HasWebsite { - stats.WebsiteCount++ - } - if fields.HasTelegram { - stats.TelegramCount++ - } -} - -func printCoverageStats() { - coverageStats.mu.Lock() - defer coverageStats.mu.Unlock() - - timestamp := time.Now().UTC().Format("2006-01-02 15:04:05") - - fmt.Printf("\n") - fmt.Printf("╔══════════════════════════════════════════════════════════════════════════════╗\n") - fmt.Printf("║ METADATA COVERAGE STATS - %s ║\n", timestamp) - fmt.Printf("╠══════════════════════════════════════════════════════════════════════════════╣\n") - fmt.Printf("║ Provider │ Checks │ Logo │ Name │ Symbol│ Desc │Twitter│Website│Telegram│ Errors │\n") - fmt.Printf("╠══════════════════════════════════════════════════════════════════════════════╣\n") - - for _, stats := range []*ProviderCoverage{&coverageStats.Mobula, &coverageStats.Codex, &coverageStats.Jupiter} { - if stats.TotalChecks == 0 { - fmt.Printf("║ %-8s │ %6d │ - │ - │ - │ - │ - │ - │ - │ %6d ║\n", - stats.Provider, stats.TotalChecks, stats.ErrorCount) - continue - } - - successChecks := stats.TotalChecks - stats.ErrorCount - if successChecks == 0 { - successChecks = 1 // Avoid division by zero - } - - fmt.Printf("║ %-8s │ %6d │ %5.1f%%│ %5.1f%%│ %5.1f%%│ %5.1f%%│ %5.1f%%│ %5.1f%%│ %5.1f%% │ %6d ║\n", - stats.Provider, - stats.TotalChecks, - float64(stats.LogoCount)/float64(successChecks)*100, - float64(stats.NameCount)/float64(successChecks)*100, - float64(stats.SymbolCount)/float64(successChecks)*100, - float64(stats.DescCount)/float64(successChecks)*100, - float64(stats.TwitterCount)/float64(successChecks)*100, - float64(stats.WebsiteCount)/float64(successChecks)*100, - float64(stats.TelegramCount)/float64(successChecks)*100, - stats.ErrorCount, - ) - } - - fmt.Printf("╚══════════════════════════════════════════════════════════════════════════════╝\n") - fmt.Printf("\n") - - coverageStats.LastPrint = time.Now() -} - -func chainNameFromPulseChainID(chainID string) string { - switch chainID { - case "solana:solana": - return "solana" - case "evm:56": - return "bnb" - case "evm:8453": - return "base" - case "evm:143": - return "monad" - default: - return chainID - } -} - -func checkTokenMetadata(token TokenToCheck, config *Config) { - chainName := chainNameFromPulseChainID(token.ChainID) - - // Check Mobula - mobulaResult := checkMobulaMetadata(token, config.MobulaAPIKey) - updateStats("mobula", mobulaResult) - - // Record Prometheus metrics for Mobula - RecordMetadataCoverage("mobula", chainName, "logo", mobulaResult.HasLogo, config.MonitorRegion) - RecordMetadataCoverage("mobula", chainName, "description", mobulaResult.HasDescription, config.MonitorRegion) - RecordMetadataCoverage("mobula", chainName, "twitter", mobulaResult.HasTwitter, config.MonitorRegion) - RecordMetadataCoverage("mobula", chainName, "website", mobulaResult.HasWebsite, config.MonitorRegion) - RecordMetadataLatency("mobula", chainName, mobulaResult.ResponseTimeMs, config.MonitorRegion) - - // Check Codex - codexResult := checkCodexMetadata(token, config.DefinedSessionCookie) - updateStats("codex", codexResult) - - // Record Prometheus metrics for Codex - RecordMetadataCoverage("codex", chainName, "logo", codexResult.HasLogo, config.MonitorRegion) - RecordMetadataCoverage("codex", chainName, "description", codexResult.HasDescription, config.MonitorRegion) - RecordMetadataCoverage("codex", chainName, "twitter", codexResult.HasTwitter, config.MonitorRegion) - RecordMetadataCoverage("codex", chainName, "website", codexResult.HasWebsite, config.MonitorRegion) - RecordMetadataLatency("codex", chainName, codexResult.ResponseTimeMs, config.MonitorRegion) - - // Check Jupiter (Solana only - scraping frontend) - var jupiterResult MetadataFields - if token.ChainID == "solana" || token.ChainID == "solana:solana" { - jupiterResult = checkJupiterMetadata(token) - updateStats("jupiter", jupiterResult) - - // Record Prometheus metrics for Jupiter - RecordMetadataCoverage("jupiter", chainName, "logo", jupiterResult.HasLogo, config.MonitorRegion) - RecordMetadataCoverage("jupiter", chainName, "description", jupiterResult.HasDescription, config.MonitorRegion) - RecordMetadataCoverage("jupiter", chainName, "twitter", jupiterResult.HasTwitter, config.MonitorRegion) - RecordMetadataCoverage("jupiter", chainName, "website", jupiterResult.HasWebsite, config.MonitorRegion) - RecordMetadataLatency("jupiter", chainName, jupiterResult.ResponseTimeMs, config.MonitorRegion) - } - - // Single condensed log line - boolToIcon := func(b bool) string { - if b { - return "✓" - } - return "✗" - } - - jupiterLogo := "-" - if token.ChainID == "solana" || token.ChainID == "solana:solana" { - jupiterLogo = boolToIcon(jupiterResult.HasLogo) - } - - fmt.Printf("[META] %s/%s | M:%s%s%s | C:%s%s%s | J:%s\n", - token.Symbol, chainName, - boolToIcon(mobulaResult.HasLogo), boolToIcon(mobulaResult.HasDescription), boolToIcon(mobulaResult.HasTwitter), - boolToIcon(codexResult.HasLogo), boolToIcon(codexResult.HasDescription), boolToIcon(codexResult.HasTwitter), - jupiterLogo) - - // Print stats every 50 checks (reduced from 10) - coverageStats.mu.Lock() - totalChecks := coverageStats.Mobula.TotalChecks - coverageStats.mu.Unlock() - - if totalChecks > 0 && totalChecks%50 == 0 { - printCoverageStats() - } -} - -// QueueTokenForMetadataCheck adds a token to the check queue -func QueueTokenForMetadataCheck(token TokenToCheck) { - select { - case tokenQueue <- token: - // Token queued successfully - default: - // Queue full, skip this token - fmt.Printf("[METADATA] Queue full, skipping token: %s\n", token.Address) - } -} - -// runMetadataCoverageMonitor starts the metadata coverage monitoring -func runMetadataCoverageMonitor(config *Config, stopChan <-chan struct{}) { - fmt.Println("Starting Metadata Coverage Monitor...") - fmt.Println(" Comparing metadata coverage: Mobula vs Codex vs Jupiter") - fmt.Println(" Fields tracked: Logo, Name, Symbol, Description, Twitter, Website, Telegram") - fmt.Println(" Note: Jupiter only supports Solana and has no description/socials") - fmt.Println(" Waiting for new tokens from Pulse stream...") - fmt.Println() - - // Stats printer ticker - print every 5 minutes - statsTicker := time.NewTicker(5 * time.Minute) - defer statsTicker.Stop() - - for { - select { - case <-stopChan: - fmt.Println("Metadata Coverage monitor stopped") - printCoverageStats() // Print final stats - return - - case token := <-tokenQueue: - // Small delay to let the token get indexed - time.Sleep(2 * time.Second) - checkTokenMetadata(token, config) - - case <-statsTicker.C: - printCoverageStats() - } - } -} - diff --git a/harnesses/metadata-coverage/cmd/script/metrics.go b/harnesses/metadata-coverage/cmd/script/metrics.go deleted file mode 100644 index 288e1b16..00000000 --- a/harnesses/metadata-coverage/cmd/script/metrics.go +++ /dev/null @@ -1,421 +0,0 @@ -package main - -import ( - "fmt" - "net/http" - "sync" - - "github.com/prometheus/client_golang/prometheus" - "github.com/prometheus/client_golang/prometheus/promhttp" -) - -var ( - // Pool discovery latency metric - poolDiscoveryLatency *prometheus.GaugeVec - poolDiscoveryErrors *prometheus.CounterVec - - // REST API latency metrics - restAPILatency *prometheus.HistogramVec - restAPIErrors *prometheus.CounterVec - restAPIStatusCodes *prometheus.CounterVec - - // Quote API latency metrics - quoteAPILatency *prometheus.HistogramVec - quoteAPIErrors *prometheus.CounterVec - quoteAPIStatusCodes *prometheus.CounterVec - - // Metadata coverage metrics - metadataCoverageTotal *prometheus.CounterVec - metadataCoverageSuccess *prometheus.CounterVec - metadataAPILatency *prometheus.HistogramVec - - // Head lag metrics - headLagBlocks *prometheus.GaugeVec - headLagSeconds *prometheus.GaugeVec - blockchainHead *prometheus.GaugeVec - aggregatorHead *prometheus.GaugeVec - headLagErrors *prometheus.CounterVec - - // Fast-trade latency (for comparison with Pulse V2) - fastTradeLatency *prometheus.GaugeVec - - // Mobula detailed head lag (with breakdown) - mobulaHeadLagDetailed *prometheus.GaugeVec - mobulaProcessingLagSeconds *prometheus.GaugeVec - mobulaNetworkLagSeconds *prometheus.GaugeVec - - // Latest tx_hash seen per pool. Kept at most 1 series per (chain, pool_address) - // via explicit Delete of the previous label set in RecordMobulaLastTx. - // Alert annotations query this gauge to include a clickable tx link. - mobulaLastTxHash *prometheus.GaugeVec - mobulaLastTxMu sync.Mutex - mobulaLastTxSeen = make(map[string]string) // key: "chain:pool_address" -> last tx_hash -) - -func init() { - poolDiscoveryLatency = prometheus.NewGaugeVec( - prometheus.GaugeOpts{ - Name: "pool_discovery_latency_milliseconds", - Help: "Time from pool creation on-chain to first trade detection (pool discovery latency)", - }, - []string{"aggregator", "chain", "region"}, - ) - prometheus.MustRegister(poolDiscoveryLatency) - - poolDiscoveryErrors = prometheus.NewCounterVec( - prometheus.CounterOpts{ - Name: "pool_discovery_errors_total", - Help: "Total number of errors when fetching pool discovery data", - }, - []string{"aggregator", "error_type", "region"}, - ) - prometheus.MustRegister(poolDiscoveryErrors) - - // REST API latency histogram with buckets optimized for API response times - restAPILatency = prometheus.NewHistogramVec( - prometheus.HistogramOpts{ - Name: "rest_api_latency_milliseconds", - Help: "REST API response latency in milliseconds", - Buckets: []float64{50, 100, 200, 500, 1000, 2000, 5000, 10000}, - }, - []string{"aggregator", "endpoint", "chain", "region"}, - ) - prometheus.MustRegister(restAPILatency) - - // REST API errors counter - restAPIErrors = prometheus.NewCounterVec( - prometheus.CounterOpts{ - Name: "rest_api_errors_total", - Help: "Total number of REST API errors", - }, - []string{"aggregator", "endpoint", "chain", "error_type", "region"}, - ) - prometheus.MustRegister(restAPIErrors) - - // REST API status codes counter - restAPIStatusCodes = prometheus.NewCounterVec( - prometheus.CounterOpts{ - Name: "rest_api_status_codes_total", - Help: "Total count of REST API responses by status code", - }, - []string{"aggregator", "endpoint", "chain", "status_code", "region"}, - ) - prometheus.MustRegister(restAPIStatusCodes) - - // Quote API latency histogram - quoteAPILatency = prometheus.NewHistogramVec( - prometheus.HistogramOpts{ - Name: "quote_api_latency_milliseconds", - Help: "Quote API response latency in milliseconds", - Buckets: []float64{50, 100, 200, 300, 500, 750, 1000, 1500, 2000, 3000, 5000}, - }, - []string{"provider", "chain", "region"}, - ) - prometheus.MustRegister(quoteAPILatency) - - // Quote API errors counter - quoteAPIErrors = prometheus.NewCounterVec( - prometheus.CounterOpts{ - Name: "quote_api_errors_total", - Help: "Total number of Quote API errors", - }, - []string{"provider", "chain", "error_type", "region"}, - ) - prometheus.MustRegister(quoteAPIErrors) - - // Quote API status codes counter - quoteAPIStatusCodes = prometheus.NewCounterVec( - prometheus.CounterOpts{ - Name: "quote_api_status_codes_total", - Help: "Total count of Quote API responses by status code", - }, - []string{"provider", "chain", "status_code", "region"}, - ) - prometheus.MustRegister(quoteAPIStatusCodes) - - // Metadata coverage - total checks per provider/chain/field - metadataCoverageTotal = prometheus.NewCounterVec( - prometheus.CounterOpts{ - Name: "metadata_coverage_checks_total", - Help: "Total number of metadata coverage checks", - }, - []string{"provider", "chain", "field", "region"}, - ) - prometheus.MustRegister(metadataCoverageTotal) - - // Metadata coverage - successful (field present) checks - metadataCoverageSuccess = prometheus.NewCounterVec( - prometheus.CounterOpts{ - Name: "metadata_coverage_success_total", - Help: "Total number of successful metadata coverage checks (field present)", - }, - []string{"provider", "chain", "field", "region"}, - ) - prometheus.MustRegister(metadataCoverageSuccess) - - // Metadata API latency - metadataAPILatency = prometheus.NewHistogramVec( - prometheus.HistogramOpts{ - Name: "metadata_api_latency_milliseconds", - Help: "Metadata API response latency in milliseconds", - Buckets: []float64{50, 100, 200, 500, 1000, 2000, 5000, 10000}, - }, - []string{"provider", "chain", "region"}, - ) - prometheus.MustRegister(metadataAPILatency) - - // Head lag - milliseconds behind (raw value) - headLagBlocks = prometheus.NewGaugeVec( - prometheus.GaugeOpts{ - Name: "head_lag_milliseconds", - Help: "Indexation latency in milliseconds (time between on-chain event and WebSocket receipt)", - }, - []string{"aggregator", "chain", "region"}, - ) - prometheus.MustRegister(headLagBlocks) - - // Head lag - seconds behind (converted from ms) - headLagSeconds = prometheus.NewGaugeVec( - prometheus.GaugeOpts{ - Name: "head_lag_seconds", - Help: "Indexation latency in seconds (time between on-chain event and WebSocket receipt)", - }, - []string{"aggregator", "chain", "region"}, - ) - prometheus.MustRegister(headLagSeconds) - - // Blockchain head block number (source of truth) - blockchainHead = prometheus.NewGaugeVec( - prometheus.GaugeOpts{ - Name: "blockchain_head_block", - Help: "Latest block number on the blockchain (source of truth)", - }, - []string{"chain", "region"}, - ) - prometheus.MustRegister(blockchainHead) - - // Aggregator head block number (what they have indexed) - aggregatorHead = prometheus.NewGaugeVec( - prometheus.GaugeOpts{ - Name: "aggregator_head_block", - Help: "Latest block number indexed by the aggregator", - }, - []string{"aggregator", "chain", "region"}, - ) - prometheus.MustRegister(aggregatorHead) - - // Head lag errors counter - headLagErrors = prometheus.NewCounterVec( - prometheus.CounterOpts{ - Name: "head_lag_errors_total", - Help: "Total number of errors when fetching head lag data", - }, - []string{"aggregator", "chain", "error_type", "region"}, - ) - prometheus.MustRegister(headLagErrors) - - // Fast-trade latency (separate from head_lag for comparison) - fastTradeLatency = prometheus.NewGaugeVec( - prometheus.GaugeOpts{ - Name: "fast_trade_latency_milliseconds", - Help: "Fast-trade WebSocket latency in milliseconds (for comparison with Pulse V2)", - }, - []string{"aggregator", "chain", "region"}, - ) - prometheus.MustRegister(fastTradeLatency) - - // Mobula head lag detailed (fixed-cardinality; breakdown exposed as separate gauges) - mobulaHeadLagDetailed = prometheus.NewGaugeVec( - prometheus.GaugeOpts{ - Name: "mobula_head_lag_detailed_seconds", - Help: "Mobula total head lag in seconds (on-chain -> WebSocket receipt)", - }, - []string{"aggregator", "chain", "region", "pool_address"}, - ) - prometheus.MustRegister(mobulaHeadLagDetailed) - - mobulaProcessingLagSeconds = prometheus.NewGaugeVec( - prometheus.GaugeOpts{ - Name: "mobula_processing_lag_seconds", - Help: "Mobula processing latency (on-chain -> Mobula processed)", - }, - []string{"aggregator", "chain", "region", "pool_address"}, - ) - prometheus.MustRegister(mobulaProcessingLagSeconds) - - mobulaNetworkLagSeconds = prometheus.NewGaugeVec( - prometheus.GaugeOpts{ - Name: "mobula_network_lag_seconds", - Help: "Network latency Mobula processed -> WebSocket receipt", - }, - []string{"aggregator", "chain", "region", "pool_address"}, - ) - prometheus.MustRegister(mobulaNetworkLagSeconds) - - // Gauge value is always 1; tx_hash is carried as a label so alert templates - // can query the latest tx per pool. Exactly ONE series per (chain, pool_address) - // is kept alive at any time — see RecordMobulaLastTx for the delete/set dance. - mobulaLastTxHash = prometheus.NewGaugeVec( - prometheus.GaugeOpts{ - Name: "mobula_last_tx_hash", - Help: "Latest Mobula trade tx_hash per pool (value=1, tx_hash label rotated)", - }, - []string{"aggregator", "chain", "region", "pool_address", "tx_hash"}, - ) - prometheus.MustRegister(mobulaLastTxHash) -} - -// RecordMobulaLastTx overwrites the single live series for (chain, pool_address) -// with a new tx_hash label. Prevents cardinality leak by deleting the previous -// label set before setting the new one. -func RecordMobulaLastTx(chain, region, poolAddress, txHash string) { - if txHash == "" { - return - } - key := chain + ":" + poolAddress - mobulaLastTxMu.Lock() - defer mobulaLastTxMu.Unlock() - - if prev, ok := mobulaLastTxSeen[key]; ok && prev != txHash { - mobulaLastTxHash.DeleteLabelValues("mobula", chain, region, poolAddress, prev) - } - mobulaLastTxHash.WithLabelValues("mobula", chain, region, poolAddress, txHash).Set(1) - mobulaLastTxSeen[key] = txHash -} - -func RecordPoolDiscoveryLatency(aggregator string, chain string, latencyMs float64, region string) { - // Filter out invalid values: negative or > 2 minutes (120000ms) - if latencyMs < 0 || latencyMs > 120000 { - return - } - - poolDiscoveryLatency.WithLabelValues(aggregator, chain, region).Set(latencyMs) -} - -// RecordPoolDiscoveryError records an error when fetching pool discovery data -func RecordPoolDiscoveryError(aggregator string, errorType string, region string) { - poolDiscoveryErrors.WithLabelValues(aggregator, errorType, region).Inc() -} - -// RecordRESTLatency records the latency of a REST API call -func RecordRESTLatency(aggregator string, endpoint string, chain string, latencyMs float64, statusCode int, region string) { - // Record latency in histogram - restAPILatency.WithLabelValues(aggregator, endpoint, chain, region).Observe(latencyMs) - - // Record status code - restAPIStatusCodes.WithLabelValues(aggregator, endpoint, chain, fmt.Sprintf("%d", statusCode), region).Inc() -} - -// RecordRESTError records a REST API error -func RecordRESTError(aggregator string, endpoint string, chain string, errorType string, region string) { - restAPIErrors.WithLabelValues(aggregator, endpoint, chain, errorType, region).Inc() -} - -// RecordQuoteAPILatency records the latency of a Quote API call -func RecordQuoteAPILatency(provider string, chain string, latencyMs float64, statusCode int, region string) { - // Record latency in histogram - quoteAPILatency.WithLabelValues(provider, chain, region).Observe(latencyMs) - - // Record status code - quoteAPIStatusCodes.WithLabelValues(provider, chain, fmt.Sprintf("%d", statusCode), region).Inc() -} - -// RecordQuoteAPIError records a Quote API error -func RecordQuoteAPIError(provider string, chain string, errorType string, region string) { - quoteAPIErrors.WithLabelValues(provider, chain, errorType, region).Inc() -} - -// RecordMetadataCoverage records metadata coverage for a specific field -func RecordMetadataCoverage(provider string, chain string, field string, present bool, region string) { - metadataCoverageTotal.WithLabelValues(provider, chain, field, region).Inc() - if present { - metadataCoverageSuccess.WithLabelValues(provider, chain, field, region).Inc() - } -} - -// RecordMetadataLatency records the latency of a metadata API call -func RecordMetadataLatency(provider string, chain string, latencyMs float64, region string) { - metadataAPILatency.WithLabelValues(provider, chain, region).Observe(latencyMs) -} - -// RecordHeadLag records the head lag for an aggregator on a specific chain -func RecordHeadLag(aggregator string, chain string, lagBlocks int64, lagSeconds float64, region string, txHash string) { - // Filter out aberrant values (> 30s likely means connection issues, not real lag) - if lagSeconds < 0 || lagSeconds > 30 { - return - } - - headLagBlocks.WithLabelValues(aggregator, chain, region).Set(float64(lagBlocks)) - headLagSeconds.WithLabelValues(aggregator, chain, region).Set(lagSeconds) - // tx_hash is logged but not stored as a metric label to avoid cardinality explosion -} - -// RecordBlockchainHead records the current blockchain head block number -func RecordBlockchainHead(chain string, blockNumber int64, region string) { - blockchainHead.WithLabelValues(chain, region).Set(float64(blockNumber)) -} - -// RecordAggregatorHead records the aggregator's indexed head block number -func RecordAggregatorHead(aggregator string, chain string, blockNumber int64, region string) { - aggregatorHead.WithLabelValues(aggregator, chain, region).Set(float64(blockNumber)) -} - -// RecordHeadLagError records an error when fetching head lag data -func RecordHeadLagError(aggregator string, chain string, errorType string, region string) { - headLagErrors.WithLabelValues(aggregator, chain, errorType, region).Inc() -} - -// RecordCodexBlockNumber records the block number from Codex events -func RecordCodexBlockNumber(chain string, blockNumber int64, region string) { - aggregatorHead.WithLabelValues("codex", chain, region).Set(float64(blockNumber)) -} - -// RecordFastTradeLatency records fast-trade WebSocket latency -func RecordFastTradeLatency(aggregator string, chain string, latencyMs float64, region string) { - // Filter out invalid values - if latencyMs < 0 || latencyMs > 120000 { - return - } - fastTradeLatency.WithLabelValues(aggregator, chain, region).Set(latencyMs) -} - -// RecordMobulaHeadLagDetailed records Mobula head lag with breakdown on fixed-cardinality gauges -func RecordMobulaHeadLagDetailed( - chain string, - region string, - poolAddress string, - txHash string, - totalLagMs int64, - mobulaProcessingMs int64, - networkLatencyMs int64, - onChainTime string, - mobulaTime string, - receivedTime string, -) { - // Drop replays/clock-skew: > 30s is not a real indexation latency - if totalLagMs < 0 || totalLagMs > 30000 { - return - } - - mobulaHeadLagDetailed.WithLabelValues("mobula", chain, region, poolAddress). - Set(float64(totalLagMs) / 1000.0) - mobulaProcessingLagSeconds.WithLabelValues("mobula", chain, region, poolAddress). - Set(float64(mobulaProcessingMs) / 1000.0) - mobulaNetworkLagSeconds.WithLabelValues("mobula", chain, region, poolAddress). - Set(float64(networkLatencyMs) / 1000.0) -} - -func StartMetricsServer(addr string) error { - mux := http.NewServeMux() - - // Prometheus metrics endpoint - mux.Handle("/metrics", promhttp.Handler()) - - // Admin cleanup endpoint - setupCleanupEndpoint(mux) - - // Debug: tail of in-memory log ring - setupLogsEndpoint(mux) - - return http.ListenAndServe(addr, mux) -} diff --git a/harnesses/metadata-coverage/cmd/script/mobula_fast_trade_monitor.go b/harnesses/metadata-coverage/cmd/script/mobula_fast_trade_monitor.go deleted file mode 100644 index 9629ef2e..00000000 --- a/harnesses/metadata-coverage/cmd/script/mobula_fast_trade_monitor.go +++ /dev/null @@ -1,225 +0,0 @@ -package main - -import ( - "encoding/json" - "fmt" - "time" - - _ "github.com/gorilla/websocket" // Used by proxy.go getProxyDialer -) - -// ============================================================================ -// Mobula Fast-Trade Monitor (separate from head lag) -// Monitors fast-trade latency for comparison with Pulse V2 -// ============================================================================ - -type MobulaFastTradeEvent struct { - Blockchain string `json:"blockchain"` - Date int64 `json:"date"` // On-chain timestamp (ms) - Timestamp int64 `json:"timestamp"` // Mobula processing timestamp (ms) - Hash string `json:"hash"` - Pair string `json:"pair"` - Address string `json:"address"` // Pool address - Type string `json:"type"` - TokenPrice float64 `json:"tokenPrice"` -} - -// Monitored pools (same chains as Pulse V2) -var fastTradePools = []struct { - Blockchain string - Address string - ChainName string -}{ - { - Blockchain: "solana", - Address: "7qbRF6YsyGuLUVs6Y1q64bdVrfe4ZcUUz1JRdoVNUJnm", // SOL/USDC Raydium - ChainName: "solana", - }, - { - Blockchain: "evm:8453", - Address: "0x4c36388be6f416a29c8d8eee81c771ce6be14b18", // WETH/USDC Base - ChainName: "base", - }, - { - Blockchain: "evm:56", - Address: "0x58f876857a02d6762e0101bb5c46a8c1ed44dc16", // WBNB/BUSD PancakeSwap - ChainName: "bnb", - }, -} - -func runMobulaFastTradeMonitor(config *Config, stopChan <-chan struct{}) { - fmt.Println() - fmt.Println("╔═══════════════════════════════════════════════════════════╗") - fmt.Println("║ MOBULA FAST-TRADE MONITOR (Comparison) ║") - fmt.Println("╠═══════════════════════════════════════════════════════════╣") - fmt.Println("║ Purpose: Compare with Pulse V2 discovery latency ║") - fmt.Println("║ Monitoring: High-volume pools on 4 chains ║") - fmt.Printf("║ Pools: %d monitored pools ║\n", len(fastTradePools)) - fmt.Println("╚═══════════════════════════════════════════════════════════╝") - fmt.Println() - - if config.MobulaAPIKey == "" { - fmt.Println("[FAST-TRADE][MOBULA] API key not set, skipping") - return - } - - reconnectDelay := 5 * time.Second - maxReconnectDelay := 60 * time.Second - - for { - select { - case <-stopChan: - fmt.Println("[FAST-TRADE][MOBULA] Monitor stopped") - return - default: - err := connectAndMonitorFastTrade(config, stopChan) - if err != nil { - fmt.Printf("[FAST-TRADE][MOBULA] Connection error: %v. Reconnecting in %v...\n", err, reconnectDelay) - - select { - case <-stopChan: - return - case <-time.After(reconnectDelay): - reconnectDelay = reconnectDelay * 2 - if reconnectDelay > maxReconnectDelay { - reconnectDelay = maxReconnectDelay - } - } - } else { - reconnectDelay = 5 * time.Second - } - } - } -} - -func connectAndMonitorFastTrade(config *Config, stopChan <-chan struct{}) error { - conn, _, err := getProxyDialer().Dial(config.MobulaWSURL, nil) - if err != nil { - return fmt.Errorf("dial failed: %w", err) - } - defer conn.Close() - - // Build subscription items - var items []map[string]interface{} - for _, pool := range fastTradePools { - items = append(items, map[string]interface{}{ - "blockchain": pool.Blockchain, - "address": pool.Address, - }) - } - - // Subscribe to fast-trade - subscribeMsg := map[string]interface{}{ - "type": "fast-trade", - "authorization": config.MobulaAPIKey, - "payload": map[string]interface{}{ - "assetMode": false, - "items": items, - }, - } - - if err := conn.WriteJSON(subscribeMsg); err != nil { - return fmt.Errorf("subscribe failed: %w", err) - } - - fmt.Printf("[FAST-TRADE][MOBULA] Subscribed to %d pools\n", len(items)) - - // Start ping goroutine - pingDone := make(chan struct{}) - go func() { - ticker := time.NewTicker(25 * time.Second) - defer ticker.Stop() - for { - select { - case <-pingDone: - return - case <-ticker.C: - if err := conn.WriteJSON(map[string]string{"event": "ping"}); err != nil { - return - } - } - } - }() - defer close(pingDone) - - // Read messages - for { - select { - case <-stopChan: - return nil - default: - conn.SetReadDeadline(time.Now().Add(60 * time.Second)) - _, message, err := conn.ReadMessage() - if err != nil { - return fmt.Errorf("read failed: %w", err) - } - - // Parse message - var trade MobulaFastTradeEvent - if err := json.Unmarshal(message, &trade); err != nil { - continue - } - - // Skip non-trade messages (pong, etc) - if trade.Hash == "" || trade.Date == 0 { - continue - } - - // Calculate fast-trade latency - receiveTime := time.Now().UTC() - onChainTime := time.UnixMilli(trade.Date) - mobulaProcessTime := time.UnixMilli(trade.Timestamp) - - // Total latency: on-chain → WebSocket receipt - totalLagMs := receiveTime.Sub(onChainTime).Milliseconds() - - // Mobula processing latency: on-chain → Mobula processed - mobulaLagMs := mobulaProcessTime.Sub(onChainTime).Milliseconds() - - // Network latency: Mobula processed → WebSocket receipt - networkLagMs := receiveTime.Sub(mobulaProcessTime).Milliseconds() - - lagSeconds := float64(totalLagMs) / 1000.0 - - // Get chain name - chainName := getChainNameFromBlockchainFastTrade(trade.Blockchain) - - // Record metric with "fast-trade" source label - RecordFastTradeLatency("mobula", chainName, float64(totalLagMs), config.MonitorRegion) - - // Enhanced logging for spikes - if totalLagMs > 3000 { - timestamp := receiveTime.Format("15:04:05") - fmt.Printf("[FAST-TRADE][MOBULA][%s][%s] 🚨 SPIKE DETECTED!\n", timestamp, chainName) - fmt.Printf(" Total Latency: %.2fs (%.0fms)\n", lagSeconds, float64(totalLagMs)) - fmt.Printf(" ├─ Mobula Processing: %.0fms (on-chain → Mobula)\n", float64(mobulaLagMs)) - fmt.Printf(" └─ Network Latency: %.0fms (Mobula → WebSocket)\n", float64(networkLagMs)) - fmt.Printf(" Tx: %s\n", trade.Hash) - fmt.Printf(" Pool: %s\n", trade.Address) - fmt.Printf(" On-chain time: %s\n", onChainTime.Format("15:04:05.000")) - fmt.Printf(" Mobula processed: %s\n", mobulaProcessTime.Format("15:04:05.000")) - fmt.Printf(" Received: %s\n", receiveTime.Format("15:04:05.000")) - } else if time.Now().Second()%30 == 0 { - // Log normal latency occasionally - timestamp := receiveTime.Format("15:04:05") - fmt.Printf("[FAST-TRADE][MOBULA][%s][%s] Latency: %.2fs | Tx: %s\n", - timestamp, chainName, lagSeconds, trade.Hash[:12]+"...") - } - } - } -} - -func getChainNameFromBlockchainFastTrade(blockchain string) string { - switch blockchain { - case "Ethereum", "evm:1": - return "ethereum" - case "Solana", "solana": - return "solana" - case "Base", "evm:8453": - return "base" - case "BNB Smart Chain (BEP20)", "BSC", "evm:56": - return "bnb" - default: - return blockchain - } -} diff --git a/harnesses/metadata-coverage/cmd/script/mobula_pulse_monitor.go b/harnesses/metadata-coverage/cmd/script/mobula_pulse_monitor.go deleted file mode 100644 index 7cc85ab9..00000000 --- a/harnesses/metadata-coverage/cmd/script/mobula_pulse_monitor.go +++ /dev/null @@ -1,190 +0,0 @@ -package main - -// Slim Mobula Pulse V2 feeder. -// -// Pulse was removed as a benched aggregator on 2026-04-28 (commit e7f2276ca5) -// because OpenChainBench dropped it from scope. But the metadata coverage -// benchmark depends on Pulse to discover NEW tokens — without that source, -// the metadata_coverage_* gauges stop receiving fresh samples and the dashboard -// goes blank. -// -// This file restores ONLY the data flow into the metadata queue. It does NOT -// emit any pulse-specific Prometheus metrics (no RecordPoolDiscoveryLatency, -// no head_lag_seconds{aggregator="pulse"}). Pulse therefore never re-appears -// in the head_lag dashboard. - -import ( - "encoding/json" - "fmt" - "log" - "time" -) - -const mobulaPulseWSURL = "wss://pulse-v2-api.mobula.io" - -// Chains we care about for token discovery (same launchpad chains as before). -var pulseChains = []string{ - "solana:solana", - "evm:56", - "evm:8453", -} - -// Source names emitted by Mobula Pulse V2 we treat as "interesting" launchpads. -// Filtering keeps the metadata queue focused on freshly-launched tokens -// (where coverage differences between providers are most visible) instead of -// every Uniswap/Raydium pool creation. -var pulseLaunchpadSources = map[string]bool{ - "pumpfun": true, - "pump.fun": true, - "meteora": true, - "meteora-dbc": true, - "meteoradbc": true, - "fourmeme": true, - "four.meme": true, - "flap": true, - "zora": true, - "baseapp": true, - "bags": true, - "moonshot": true, - "raydium_cpmm": true, -} - -type pulseSubscribePayload struct { - Model string `json:"model"` - AssetMode bool `json:"assetMode"` - ChainID []string `json:"chainId"` - Compressed bool `json:"compressed"` - Views []map[string]any `json:"views,omitempty"` -} - -type pulseSubscribeMsg struct { - Type string `json:"type"` - Authorization string `json:"authorization"` - Payload pulseSubscribePayload `json:"payload"` -} - -type pulseToken struct { - Address string `json:"address"` - Name string `json:"name"` - Symbol string `json:"symbol"` - ChainID string `json:"chainId"` - Source string `json:"source"` - CreatedAt string `json:"createdAt"` -} - -type pulseTokenWrapper struct { - Token pulseToken `json:"token"` - Source string `json:"source"` -} - -type pulseNewTokenMsg struct { - Type string `json:"type"` - Payload struct { - ViewName string `json:"viewName"` - Token pulseTokenWrapper `json:"token"` - Source string `json:"source"` - } `json:"payload"` -} - -func runMobulaPulseMonitor(config *Config, stopChan <-chan struct{}) { - if config.MobulaAPIKey == "" { - fmt.Println("[PULSE-FEEDER] MOBULA_API_KEY not set — skipping pulse feeder, metadata coverage queue will stay empty.") - return - } - - fmt.Println("[PULSE-FEEDER] Starting (feeds metadata coverage queue only — no head_lag metrics).") - - reconnectDelay := 5 * time.Second - const maxReconnectDelay = 60 * time.Second - - for { - select { - case <-stopChan: - fmt.Println("[PULSE-FEEDER] Stopped.") - return - default: - err := pulseFeederSession(config, stopChan) - if err != nil { - log.Printf("[PULSE-FEEDER] session error: %v — reconnect in %v", err, reconnectDelay) - } - select { - case <-stopChan: - return - case <-time.After(reconnectDelay): - reconnectDelay *= 2 - if reconnectDelay > maxReconnectDelay { - reconnectDelay = maxReconnectDelay - } - } - } - } -} - -func pulseFeederSession(config *Config, stopChan <-chan struct{}) error { - headers := map[string][]string{"Authorization": {config.MobulaAPIKey}} - dialer := getProxyDialer() - conn, _, err := dialer.Dial(mobulaPulseWSURL, headers) - if err != nil { - return fmt.Errorf("dial: %w", err) - } - defer conn.Close() - - sub := pulseSubscribeMsg{ - Type: "pulse-v2", - Authorization: config.MobulaAPIKey, - Payload: pulseSubscribePayload{ - Model: "default", - AssetMode: true, - ChainID: pulseChains, - Views: []map[string]any{ - {"name": "new", "sortBy": "created_at", "sortOrder": "desc", "limit": 50}, - }, - }, - } - if err := conn.WriteJSON(sub); err != nil { - return fmt.Errorf("subscribe: %w", err) - } - fmt.Println("[PULSE-FEEDER] subscribed; queueing launchpad tokens for metadata checks.") - - for { - select { - case <-stopChan: - return nil - default: - } - _, raw, err := conn.ReadMessage() - if err != nil { - return fmt.Errorf("read: %w", err) - } - var head struct { - Type string `json:"type"` - } - if json.Unmarshal(raw, &head) != nil || head.Type != "new-token" { - continue - } - var msg pulseNewTokenMsg - if json.Unmarshal(raw, &msg) != nil { - continue - } - - t := msg.Payload.Token.Token - src := msg.Payload.Token.Source - if src == "" { - src = msg.Payload.Source - } - if src == "" { - src = t.Source - } - if !pulseLaunchpadSources[src] { - continue - } - - QueueTokenForMetadataCheck(TokenToCheck{ - Address: t.Address, - ChainID: t.ChainID, - Symbol: t.Symbol, - Name: t.Name, - DetectedAt: time.Now().UTC(), - }) - } -} diff --git a/harnesses/metadata-coverage/cmd/script/mobula_rest_monitor.go b/harnesses/metadata-coverage/cmd/script/mobula_rest_monitor.go deleted file mode 100644 index 52315efe..00000000 --- a/harnesses/metadata-coverage/cmd/script/mobula_rest_monitor.go +++ /dev/null @@ -1,185 +0,0 @@ -package main - -import ( - "encoding/json" - "fmt" - "io" - "log" - "net/http" - "time" -) - -const ( - mobulaRESTBaseURL = "https://api.mobula.io" -) - -// Chains for REST monitoring - aligned with all monitors -var mobulaRESTChains = []struct { - blockchain string - blockchainID string - chainName string - poolAddress string -}{ - {"Solana", "solana", "solana", "7qbRF6YsyGuLUVs6Y1q64bdVrfe4ZcUUz1JRdoVNUJnm"}, // SOL/USDC - {"Base", "8453", "base", "0x4c36388be6f416a29c8d8eee81c771ce6be14b18"}, // WETH/USDC Base - {"BSC", "56", "bnb", "0x58F876857a02D6762E0101bb5C46A8c1ED44Dc16"}, // WBNB/BUSD PancakeSwap -} - -type MobulaMarketDataResponse struct { - Data []struct { - Volume float64 `json:"volume"` - Open float64 `json:"open"` - High float64 `json:"high"` - Low float64 `json:"low"` - Close float64 `json:"close"` - Time int64 `json:"time"` - } `json:"data"` -} - -// callMobulaMarketDataAPI makes a REST call to Mobula's market history/pair endpoint -func callMobulaMarketDataAPI(apiKey string, poolAddress string, blockchain string, chainName string) (float64, int, error) { - endpoint := fmt.Sprintf("%s/api/1/market/history/pair", mobulaRESTBaseURL) - - // Create HTTP client with timeout - client := &http.Client{ - Timeout: 10 * time.Second, - } - - // Build request - req, err := http.NewRequest("GET", endpoint, nil) - if err != nil { - return 0, 0, fmt.Errorf("failed to create request: %w", err) - } - - // Add query parameters - // Get last 1 hour of data with 1 minute candles - to := time.Now().UnixMilli() - from := time.Now().Add(-1 * time.Hour).UnixMilli() - - q := req.URL.Query() - q.Add("address", poolAddress) - q.Add("blockchain", blockchain) - q.Add("period", "1min") - q.Add("from", fmt.Sprintf("%d", from)) - q.Add("to", fmt.Sprintf("%d", to)) - q.Add("amount", "5") // Just get 5 candles, we don't care about data - req.URL.RawQuery = q.Encode() - - // Add headers - req.Header.Set("Authorization", apiKey) - req.Header.Set("Content-Type", "application/json") - - // Measure latency - startTime := time.Now() - resp, err := client.Do(req) - latencyMs := float64(time.Since(startTime).Milliseconds()) - - if err != nil { - return latencyMs, 0, fmt.Errorf("request failed: %w", err) - } - defer resp.Body.Close() - - // Read response body for debugging - body, _ := io.ReadAll(resp.Body) - - // Try to parse response - var marketData MobulaMarketDataResponse - if err := json.Unmarshal(body, &marketData); err != nil { - // Not a critical error, we still measured latency - log.Printf("[MOBULA-REST][%s] Response parse warning: %v (status: %d)", chainName, err, resp.StatusCode) - } - - return latencyMs, resp.StatusCode, nil -} - -// monitorMobulaREST continuously monitors Mobula REST API latency -func monitorMobulaREST(config *Config, stopChan <-chan struct{}) { - fmt.Println("Starting Mobula REST API monitor...") - fmt.Printf(" Monitoring %d chains with 20s interval\n", len(mobulaRESTChains)) - fmt.Printf(" Endpoint: /api/1/market/history/pair\n") - fmt.Println() - - if config.MobulaAPIKey == "" { - fmt.Println("MOBULA_API_KEY not set in .env file. Skipping Mobula REST monitor.") - return - } - - // Create ticker for 20 second intervals - ticker := time.NewTicker(20 * time.Second) - defer ticker.Stop() - - // Run once immediately - performMobulaRESTChecks(config) - - // Then run every 20 seconds - for { - select { - case <-stopChan: - fmt.Println("Mobula REST monitor stopped") - return - case <-ticker.C: - performMobulaRESTChecks(config) - } - } -} - -// performMobulaRESTChecks performs REST API calls to all chains -func performMobulaRESTChecks(config *Config) { - timestamp := time.Now().UTC().Format("2006-01-02 15:04:05") - - for _, chain := range mobulaRESTChains { - latencyMs, statusCode, err := callMobulaMarketDataAPI( - config.MobulaAPIKey, - chain.poolAddress, - chain.blockchainID, - chain.chainName, - ) - - if err != nil { - // Record error - errorType := "request_error" - if statusCode >= 500 { - errorType = "server_error" - } else if statusCode >= 400 { - errorType = "client_error" - } else if statusCode == 0 { - errorType = "timeout_error" - } - - RecordRESTError("mobula", "market_data", chain.chainName, errorType, config.MonitorRegion) - - fmt.Printf("[MOBULA-REST][%s][%s] ERROR | Latency: %.0fms | Status: %d | Error: %v\n", - timestamp, - chain.chainName, - latencyMs, - statusCode, - err, - ) - continue - } - - // Record successful latency measurement - RecordRESTLatency("mobula", "market_data", chain.chainName, latencyMs, statusCode, config.MonitorRegion) - - // Log the result - statusEmoji := "✓" - if statusCode >= 400 { - statusEmoji = "✗" - } else if statusCode >= 300 { - statusEmoji = "⚠" - } - - fmt.Printf("[MOBULA-REST][%s][%s] %s | Latency: %.0fms | Status: %d\n", - timestamp, - chain.chainName, - statusEmoji, - latencyMs, - statusCode, - ) - } -} - -// runMobulaRESTMonitor is the entry point for the Mobula REST monitor -func runMobulaRESTMonitor(config *Config, stopChan <-chan struct{}) { - monitorMobulaREST(config, stopChan) -} diff --git a/harnesses/metadata-coverage/cmd/script/moralis_rest_monitor.go b/harnesses/metadata-coverage/cmd/script/moralis_rest_monitor.go deleted file mode 100644 index 6a646883..00000000 --- a/harnesses/metadata-coverage/cmd/script/moralis_rest_monitor.go +++ /dev/null @@ -1,219 +0,0 @@ -package main - -import ( - "encoding/json" - "fmt" - "io" - "net/http" - "strings" - "sync" - "time" -) - -// ============================================================================ -// Moralis REST API Monitor -// Triggered by WebSocket trades to measure indexation lag -// ============================================================================ - -type MoralisOHLCVResponse struct { - PairAddress string `json:"pairAddress"` - Result []struct { - Timestamp string `json:"timestamp"` - Close float64 `json:"close"` - Volume float64 `json:"volume"` - Trades int `json:"trades"` - } `json:"result"` -} - -type MoralisMonitorPool struct { - Name string - Chain string // Chain name for metrics - ChainID string // Chain ID for Moralis API (hex for EVM, "solana" for Solana) - PairAddress string - IsEVM bool -} - -// Map WebSocket pool addresses to Moralis pairs -var moralisPairMapping = map[string]MoralisMonitorPool{ - // Solana - "7qbRF6YsyGuLUVs6Y1q64bdVrfe4ZcUUz1JRdoVNUJnm": { - Name: "SOL/USDC Raydium", - Chain: "solana", - ChainID: "solana", - PairAddress: "7qbRF6YsyGuLUVs6Y1q64bdVrfe4ZcUUz1JRdoVNUJnm", - IsEVM: false, - }, - // Base - "0x4c36388be6f416a29c8d8eee81c771ce6be14b18": { - Name: "WETH/USDC Base", - Chain: "base", - ChainID: "0x2105", - PairAddress: "0x4c36388be6f416a29c8d8eee81c771ce6be14b18", - IsEVM: true, - }, - // BNB - "0x58f876857a02d6762e0101bb5c46a8c1ed44dc16": { - Name: "WBNB/BUSD PancakeSwap", - Chain: "bnb", - ChainID: "0x38", - PairAddress: "0x58f876857a02d6762e0101bb5c46a8c1ed44dc16", - IsEVM: true, - }, -} - -var ( - moralisCheckQueue = make(chan TradeCheckRequest, 1000) - moralisHttpClient = &http.Client{Timeout: 10 * time.Second} -) - -type TradeCheckRequest struct { - PairAddress string - OnChainTime time.Time - TransactionHash string -} - -func runMoralisRESTMonitor(config *Config, stopChan <-chan struct{}, wg *sync.WaitGroup) { - defer wg.Done() - - fmt.Println("[HEAD-LAG][MORALIS-REST] Starting triggered REST monitor...") - fmt.Println("[HEAD-LAG][MORALIS-REST] Will check Moralis API when trades arrive via WebSocket") - - // Start worker to process check requests - for { - select { - case <-stopChan: - fmt.Println("[HEAD-LAG][MORALIS-REST] Monitor stopped") - return - case req := <-moralisCheckQueue: - checkMoralisForTrade(config, req) - } - } -} - -// TriggerMoralisCheck is called when a trade is detected via WebSocket -// It queues a check to see if Moralis has indexed it yet -func TriggerMoralisCheck(pairAddress string, onChainTime time.Time, txHash string) { - // Normalize address - pairAddress = strings.ToLower(pairAddress) - - // Check if we monitor this pair - if _, exists := moralisPairMapping[pairAddress]; !exists { - return - } - - select { - case moralisCheckQueue <- TradeCheckRequest{ - PairAddress: pairAddress, - OnChainTime: onChainTime, - TransactionHash: txHash, - }: - default: - // Queue full, skip - } -} - -func checkMoralisForTrade(config *Config, req TradeCheckRequest) { - pool, exists := moralisPairMapping[req.PairAddress] - if !exists { - return - } - - // Skip if no Moralis API key configured - // TODO: Add MORALIS_API_KEY to config - // For now, skip Moralis checks silently - return - - // Build URL using correct Moralis Web3 Data API - url := fmt.Sprintf("https://deep-index.moralis.io/api/v2.2/pairs/%s/ohlcv", pool.PairAddress) - - // Query from slightly before the on-chain trade to now - toDate := time.Now().UTC() - fromDate := req.OnChainTime.Add(-2 * time.Minute) // Start 2 minutes before trade - - httpReq, err := http.NewRequest("GET", url, nil) - if err != nil { - RecordHeadLagError("moralis", pool.Chain, "request_creation_failed", config.MonitorRegion) - return - } - - q := httpReq.URL.Query() - if pool.IsEVM { - q.Add("chain", pool.ChainID) - } - q.Add("to_date", fmt.Sprintf("%d", toDate.Unix())) - q.Add("from_date", fmt.Sprintf("%d", fromDate.Unix())) - q.Add("timeframe", "1m") - httpReq.URL.RawQuery = q.Encode() - - // Set headers with API key - // httpReq.Header.Set("X-API-Key", config.MoralisAPIKey) - httpReq.Header.Set("Accept", "application/json") - - // Make request - checkTime := time.Now() - resp, err := moralisHttpClient.Do(httpReq) - if err != nil { - RecordHeadLagError("moralis", pool.Chain, "request_failed", config.MonitorRegion) - return - } - defer resp.Body.Close() - - if resp.StatusCode != 200 { - RecordHeadLagError("moralis", pool.Chain, fmt.Sprintf("http_%d", resp.StatusCode), config.MonitorRegion) - return - } - - // Parse response - body, err := io.ReadAll(resp.Body) - if err != nil { - RecordHeadLagError("moralis", pool.Chain, "read_body_failed", config.MonitorRegion) - return - } - - var data MoralisOHLCVResponse - if err := json.Unmarshal(body, &data); err != nil { - RecordHeadLagError("moralis", pool.Chain, "json_parse_failed", config.MonitorRegion) - return - } - - if len(data.Result) == 0 { - // No data yet - trade not indexed - RecordHeadLagError("moralis", pool.Chain, "trade_not_found", config.MonitorRegion) - return - } - - // Find the candle that contains our on-chain trade time - // The candle timestamp is the START of the 1-minute window - tradeMinute := req.OnChainTime.Truncate(time.Minute) - - found := false - for _, candle := range data.Result { - candleTime, err := time.Parse("2006-01-02T15:04:05.000Z", candle.Timestamp) - if err != nil { - continue - } - - // Check if this candle contains our trade - // Candle at 09:05:00 contains trades from 09:05:00 to 09:05:59 - if candleTime.Equal(tradeMinute) || candleTime.Before(tradeMinute) && candleTime.Add(time.Minute).After(req.OnChainTime) { - // Found! Calculate lag - lagMs := checkTime.Sub(req.OnChainTime).Milliseconds() - lagSeconds := float64(lagMs) / 1000.0 - - // Record metrics with tx hash - RecordHeadLag("moralis", pool.Chain, lagMs, lagSeconds, config.MonitorRegion, req.TransactionHash) - - // Log - fmt.Printf("[HEAD-LAG][MORALIS][%s][%s] Trade found! Lag: %.2fs | Tx: %s | Candle: %s\n", - checkTime.Format("15:04:05"), pool.Chain, lagSeconds, req.TransactionHash[:16], candle.Timestamp) - - found = true - break - } - } - - if !found { - // Trade happened but not in any candle yet - RecordHeadLagError("moralis", pool.Chain, "trade_not_in_candles", config.MonitorRegion) - } -} diff --git a/harnesses/metadata-coverage/cmd/script/proxy.go b/harnesses/metadata-coverage/cmd/script/proxy.go deleted file mode 100644 index ac6c38aa..00000000 --- a/harnesses/metadata-coverage/cmd/script/proxy.go +++ /dev/null @@ -1,103 +0,0 @@ -package main - -import ( - "context" - "io" - "net" - "net/http" - "net/url" - "os" - "time" - - "github.com/gorilla/websocket" -) - -// getProxyDialer returns a websocket.Dialer configured with proxy from env -// IMPORTANT: Forces new connection for each dial to enable proxy IP rotation -func getProxyDialer() *websocket.Dialer { - dialer := &websocket.Dialer{ - // Force new TCP connection for each dial (no keep-alive reuse) - // This ensures rotating proxy assigns a new IP from the pool - NetDialContext: func(ctx context.Context, network, addr string) (net.Conn, error) { - netDialer := &net.Dialer{ - Timeout: 30 * time.Second, - KeepAlive: -1, // Disable keep-alive to prevent connection reuse - } - return netDialer.DialContext(ctx, network, addr) - }, - HandshakeTimeout: 30 * time.Second, - } - - // Check for proxy configuration from environment - proxyURL := os.Getenv("HTTP_PROXY") - if proxyURL == "" { - proxyURL = os.Getenv("HTTPS_PROXY") - } - - if proxyURL != "" { - parsedURL, err := url.Parse(proxyURL) - if err == nil { - dialer.Proxy = http.ProxyURL(parsedURL) - } - } - - return dialer -} - -// getProxyDialerWithSubprotocols returns a websocket.Dialer with proxy and custom subprotocols -// Each call creates a FRESH dialer to ensure proxy IP rotation works correctly -func getProxyDialerWithSubprotocols(subprotocols []string) *websocket.Dialer { - dialer := getProxyDialer() - dialer.Subprotocols = subprotocols - return dialer -} - -// getProxyHTTPClient returns an http.Client configured with proxy from env -// IMPORTANT: Creates new client for each call to enable proxy IP rotation -func getProxyHTTPClient() *http.Client { - client := &http.Client{ - Timeout: 10 * time.Second, - Transport: &http.Transport{ - // Disable keep-alive to prevent connection reuse (enables proxy rotation) - DisableKeepAlives: true, - DialContext: (&net.Dialer{ - Timeout: 30 * time.Second, - KeepAlive: -1, // Disable keep-alive - }).DialContext, - }, - } - - // Check for proxy configuration from environment - proxyURL := os.Getenv("HTTP_PROXY") - if proxyURL == "" { - proxyURL = os.Getenv("HTTPS_PROXY") - } - - if proxyURL != "" { - parsedURL, err := url.Parse(proxyURL) - if err == nil { - client.Transport.(*http.Transport).Proxy = http.ProxyURL(parsedURL) - } - } - - return client -} - -// getOutboundIP fetches the current outbound IP address through the proxy -// Used to verify proxy IP rotation is working correctly -func getOutboundIP() (string, error) { - client := getProxyHTTPClient() - - resp, err := client.Get("https://api.ipify.org?format=text") - if err != nil { - return "", err - } - defer resp.Body.Close() - - body, err := io.ReadAll(resp.Body) - if err != nil { - return "", err - } - - return string(body), nil -} diff --git a/harnesses/metadata-coverage/cmd/script/quote_api_monitor.go b/harnesses/metadata-coverage/cmd/script/quote_api_monitor.go deleted file mode 100644 index f4470f47..00000000 --- a/harnesses/metadata-coverage/cmd/script/quote_api_monitor.go +++ /dev/null @@ -1,493 +0,0 @@ -package main - -import ( - "encoding/json" - "fmt" - "io" - "net/http" - "net/url" - "time" -) - -// Quote API endpoints -const ( - // Free APIs (no API key required) - jupiterPublicURL = "https://public.jupiterapi.com/quote" // Free, 10 req/sec, Solana only - mobulaSwapURL = "https://api.mobula.io/api/2/swap/quoting" // Solana only for now - openOceanQuoteURL = "https://open-api.openocean.finance/v3" - paraSwapQuoteURL = "https://apiv5.paraswap.io/prices" - kyberSwapQuoteURL = "https://aggregator-api.kyberswap.com" - lifiQuoteURL = "https://li.quest/v1/quote" -) - -// Dummy wallet addresses for APIs that require fromAddress -const dummyWalletAddressEVM = "0xd8dA6BF26964aF9D7eEd9e03E53415D37aA96045" // Vitalik's address (EVM) -const dummyWalletAddressSolana = "HN7cABqLq46Es1jh92dQQisAq662SmxELLLsHHe4YWrH" // Random Solana wallet - -// Chain configurations for quote testing -type QuoteChainConfig struct { - Name string - ChainID string // Numeric chain ID - OpenOceanChain string // OpenOcean chain key - KyberChainKey string // KyberSwap chain key - TokenIn string // Input token address - TokenOut string // Output token address - TokenInSymbol string - TokenOutSymbol string - Amount string // Amount in smallest unit - Decimals int -} - -// Solana config for Jupiter -var solanaConfig = QuoteChainConfig{ - Name: "solana", - TokenIn: "EPjFWdd5AufqSSqeM2qN1xzybapC8G4wEGGkZwyTDt1v", // USDC - TokenOut: "So11111111111111111111111111111111111111112", // SOL - TokenInSymbol: "USDC", - TokenOutSymbol: "SOL", - Amount: "100000000", // 100 USDC (6 decimals) - Decimals: 6, -} - -// EVM chains config -var evmQuoteChains = []QuoteChainConfig{ - { - Name: "base", - ChainID: "8453", - OpenOceanChain: "8453", - KyberChainKey: "base", - TokenIn: "0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913", // USDC on Base - TokenOut: "0x4200000000000000000000000000000000000006", // WETH on Base - TokenInSymbol: "USDC", - TokenOutSymbol: "WETH", - Amount: "100000000", // 100 USDC (6 decimals) - Decimals: 6, - }, - { - Name: "bnb", - ChainID: "56", - OpenOceanChain: "56", - KyberChainKey: "bsc", - TokenIn: "0x8AC76a51cc950d9822D68b83fE1Ad97B32Cd580d", // USDC on BSC (18 decimals) - TokenOut: "0xbb4CdB9CBd36B01bD1cBaEBF2De08d9173bc095c", // WBNB - TokenInSymbol: "USDC", - TokenOutSymbol: "WBNB", - Amount: "100000000000000000000", // 100 USDC (18 decimals on BSC) - Decimals: 18, - }, -} - -// HTTP client with timeout -var quoteHTTPClient = &http.Client{ - Timeout: 15 * time.Second, -} - -// ============================================================================ -// Mobula Swap Quoting API (Solana + Base + Arbitrum, requires API key) -// ============================================================================ - -func callMobulaSwapQuoteAPI(chainID string, chainName string, tokenIn string, tokenOut string, amount string, apiKey string) (float64, int, error) { - // Use appropriate wallet address based on chain - walletAddress := dummyWalletAddressEVM - if chainName == "solana" { - walletAddress = dummyWalletAddressSolana - } - - params := url.Values{} - params.Add("chainId", chainID) - params.Add("tokenIn", tokenIn) - params.Add("tokenOut", tokenOut) - params.Add("amount", amount) - params.Add("walletAddress", walletAddress) - params.Add("slippage", "1") - - fullURL := fmt.Sprintf("%s?%s", mobulaSwapURL, params.Encode()) - - req, err := http.NewRequest("GET", fullURL, nil) - if err != nil { - return 0, 0, fmt.Errorf("failed to create request: %w", err) - } - req.Header.Set("Accept", "application/json") - if apiKey != "" { - req.Header.Set("Authorization", apiKey) - } - - startTime := time.Now() - resp, err := quoteHTTPClient.Do(req) - latencyMs := float64(time.Since(startTime).Milliseconds()) - - if err != nil { - return latencyMs, 0, fmt.Errorf("request failed: %w", err) - } - defer resp.Body.Close() - - // Read body to check for errors - body, _ := io.ReadAll(resp.Body) - - // Check for API errors in response body - var result map[string]interface{} - if err := json.Unmarshal(body, &result); err == nil { - if errMsg, ok := result["error"]; ok && errMsg != nil { - // Return 400 to indicate API error (even if HTTP was 200) - return latencyMs, 400, nil - } - } - - return latencyMs, resp.StatusCode, nil -} - -// ============================================================================ -// Jupiter Public API (Solana only, FREE - 10 req/sec) -// ============================================================================ - -func callJupiterPublicQuoteAPI() (float64, int, error) { - params := url.Values{} - params.Add("inputMint", solanaConfig.TokenIn) - params.Add("outputMint", solanaConfig.TokenOut) - params.Add("amount", solanaConfig.Amount) - params.Add("slippageBps", "50") - - fullURL := fmt.Sprintf("%s?%s", jupiterPublicURL, params.Encode()) - - req, err := http.NewRequest("GET", fullURL, nil) - if err != nil { - return 0, 0, fmt.Errorf("failed to create request: %w", err) - } - req.Header.Set("Accept", "application/json") - - startTime := time.Now() - resp, err := quoteHTTPClient.Do(req) - latencyMs := float64(time.Since(startTime).Milliseconds()) - - if err != nil { - return latencyMs, 0, fmt.Errorf("request failed: %w", err) - } - defer resp.Body.Close() - - _, _ = io.ReadAll(resp.Body) - - return latencyMs, resp.StatusCode, nil -} - -// ============================================================================ -// OpenOcean API (Multi-chain, FREE) -// ============================================================================ - -func callOpenOceanQuoteAPI(chain QuoteChainConfig) (float64, int, error) { - endpoint := fmt.Sprintf("%s/%s/quote", openOceanQuoteURL, chain.OpenOceanChain) - - params := url.Values{} - params.Add("inTokenAddress", chain.TokenIn) - params.Add("outTokenAddress", chain.TokenOut) - params.Add("amount", chain.Amount) - params.Add("gasPrice", "5") - - fullURL := fmt.Sprintf("%s?%s", endpoint, params.Encode()) - - req, err := http.NewRequest("GET", fullURL, nil) - if err != nil { - return 0, 0, fmt.Errorf("failed to create request: %w", err) - } - req.Header.Set("Accept", "application/json") - - startTime := time.Now() - resp, err := quoteHTTPClient.Do(req) - latencyMs := float64(time.Since(startTime).Milliseconds()) - - if err != nil { - return latencyMs, 0, fmt.Errorf("request failed: %w", err) - } - defer resp.Body.Close() - - _, _ = io.ReadAll(resp.Body) - - return latencyMs, resp.StatusCode, nil -} - -// ============================================================================ -// ParaSwap API (Multi-chain, FREE) -// ============================================================================ - -func callParaSwapQuoteAPI(chain QuoteChainConfig) (float64, int, error) { - params := url.Values{} - params.Add("srcToken", chain.TokenIn) - params.Add("destToken", chain.TokenOut) - params.Add("amount", chain.Amount) - params.Add("srcDecimals", fmt.Sprintf("%d", chain.Decimals)) - params.Add("destDecimals", "18") // Native tokens are 18 decimals - params.Add("network", chain.ChainID) - - fullURL := fmt.Sprintf("%s?%s", paraSwapQuoteURL, params.Encode()) - - req, err := http.NewRequest("GET", fullURL, nil) - if err != nil { - return 0, 0, fmt.Errorf("failed to create request: %w", err) - } - req.Header.Set("Accept", "application/json") - - startTime := time.Now() - resp, err := quoteHTTPClient.Do(req) - latencyMs := float64(time.Since(startTime).Milliseconds()) - - if err != nil { - return latencyMs, 0, fmt.Errorf("request failed: %w", err) - } - defer resp.Body.Close() - - _, _ = io.ReadAll(resp.Body) - - return latencyMs, resp.StatusCode, nil -} - -// ============================================================================ -// Li.Fi API (Multi-chain, FREE) -// ============================================================================ - -func callLifiQuoteAPI(chain QuoteChainConfig) (float64, int, error) { - params := url.Values{} - params.Add("fromChain", chain.ChainID) - params.Add("toChain", chain.ChainID) // Same chain swap - params.Add("fromToken", chain.TokenIn) - params.Add("toToken", chain.TokenOut) - params.Add("fromAmount", chain.Amount) - params.Add("fromAddress", dummyWalletAddressEVM) // Required by Li.Fi - - fullURL := fmt.Sprintf("%s?%s", lifiQuoteURL, params.Encode()) - - req, err := http.NewRequest("GET", fullURL, nil) - if err != nil { - return 0, 0, fmt.Errorf("failed to create request: %w", err) - } - req.Header.Set("Accept", "application/json") - - startTime := time.Now() - resp, err := quoteHTTPClient.Do(req) - latencyMs := float64(time.Since(startTime).Milliseconds()) - - if err != nil { - return latencyMs, 0, fmt.Errorf("request failed: %w", err) - } - defer resp.Body.Close() - - _, _ = io.ReadAll(resp.Body) - - return latencyMs, resp.StatusCode, nil -} - -// ============================================================================ -// KyberSwap API (Multi-chain, FREE) -// ============================================================================ - -func callKyberSwapQuoteAPI(chain QuoteChainConfig) (float64, int, error) { - endpoint := fmt.Sprintf("%s/%s/api/v1/routes", kyberSwapQuoteURL, chain.KyberChainKey) - - params := url.Values{} - params.Add("tokenIn", chain.TokenIn) - params.Add("tokenOut", chain.TokenOut) - params.Add("amountIn", chain.Amount) - - fullURL := fmt.Sprintf("%s?%s", endpoint, params.Encode()) - - req, err := http.NewRequest("GET", fullURL, nil) - if err != nil { - return 0, 0, fmt.Errorf("failed to create request: %w", err) - } - req.Header.Set("Accept", "application/json") - - startTime := time.Now() - resp, err := quoteHTTPClient.Do(req) - latencyMs := float64(time.Since(startTime).Milliseconds()) - - if err != nil { - return latencyMs, 0, fmt.Errorf("request failed: %w", err) - } - defer resp.Body.Close() - - _, _ = io.ReadAll(resp.Body) - - return latencyMs, resp.StatusCode, nil -} - - -// ============================================================================ -// Main monitoring function -// ============================================================================ - -func performQuoteAPIChecks(config *Config) { - timestamp := time.Now().UTC().Format("2006-01-02 15:04:05") - - fmt.Printf("\n[QUOTE-API][%s] === Starting quote API latency checks ===\n", timestamp) - - // ========== SOLANA QUOTES ========== - - // Mobula (Solana) - latencyMs, statusCode, err := callMobulaSwapQuoteAPI( - "solana", - "solana", - solanaConfig.TokenIn, - solanaConfig.TokenOut, - "100", // 100 USDC - config.MobulaAPIKey, - ) - if err != nil || statusCode >= 400 { - RecordQuoteAPIError("mobula", "solana", getErrorType(statusCode), config.MonitorRegion) - fmt.Printf("[QUOTE-API][%s][mobula][solana] %s | Latency: %.0fms | Status: %d\n", - timestamp, getStatusEmoji(statusCode), latencyMs, statusCode) - } else { - RecordQuoteAPILatency("mobula", "solana", latencyMs, statusCode, config.MonitorRegion) - fmt.Printf("[QUOTE-API][%s][mobula][solana] %s | Latency: %.0fms | Status: %d\n", - timestamp, getStatusEmoji(statusCode), latencyMs, statusCode) - } - - // Jupiter (Solana only - FREE public API) - latencyMs, statusCode, err = callJupiterPublicQuoteAPI() - if err != nil || statusCode >= 400 { - RecordQuoteAPIError("jupiter", "solana", getErrorType(statusCode), config.MonitorRegion) - fmt.Printf("[QUOTE-API][%s][jupiter][solana] %s | Latency: %.0fms | Status: %d\n", - timestamp, getStatusEmoji(statusCode), latencyMs, statusCode) - } else { - RecordQuoteAPILatency("jupiter", "solana", latencyMs, statusCode, config.MonitorRegion) - fmt.Printf("[QUOTE-API][%s][jupiter][solana] %s | Latency: %.0fms | Status: %d\n", - timestamp, getStatusEmoji(statusCode), latencyMs, statusCode) - } - - // ========== EVM QUOTES ========== - - // Test EVM chains with FREE APIs: Mobula (Base), OpenOcean, ParaSwap, Li.Fi, KyberSwap - for _, chain := range evmQuoteChains { - // Mobula (Base - chain where MobulaRouter is deployed) - if chain.Name == "base" { - latencyMs, statusCode, err := callMobulaSwapQuoteAPI( - "evm:"+chain.ChainID, - chain.Name, - chain.TokenIn, - chain.TokenOut, - "100", // 100 USDC - config.MobulaAPIKey, - ) - if err != nil || statusCode >= 400 { - RecordQuoteAPIError("mobula", chain.Name, getErrorType(statusCode), config.MonitorRegion) - fmt.Printf("[QUOTE-API][%s][mobula][%s] %s | Latency: %.0fms | Status: %d\n", - timestamp, chain.Name, getStatusEmoji(statusCode), latencyMs, statusCode) - } else { - RecordQuoteAPILatency("mobula", chain.Name, latencyMs, statusCode, config.MonitorRegion) - fmt.Printf("[QUOTE-API][%s][mobula][%s] %s | Latency: %.0fms | Status: %d\n", - timestamp, chain.Name, getStatusEmoji(statusCode), latencyMs, statusCode) - } - } - - // OpenOcean (FREE) - latencyMs, statusCode, err := callOpenOceanQuoteAPI(chain) - if err != nil || statusCode >= 400 { - RecordQuoteAPIError("openocean", chain.Name, getErrorType(statusCode), config.MonitorRegion) - fmt.Printf("[QUOTE-API][%s][openocean][%s] %s | Latency: %.0fms | Status: %d\n", - timestamp, chain.Name, getStatusEmoji(statusCode), latencyMs, statusCode) - } else { - RecordQuoteAPILatency("openocean", chain.Name, latencyMs, statusCode, config.MonitorRegion) - fmt.Printf("[QUOTE-API][%s][openocean][%s] %s | Latency: %.0fms | Status: %d\n", - timestamp, chain.Name, getStatusEmoji(statusCode), latencyMs, statusCode) - } - - // ParaSwap (FREE) - latencyMs, statusCode, err = callParaSwapQuoteAPI(chain) - if err != nil || statusCode >= 400 { - RecordQuoteAPIError("paraswap", chain.Name, getErrorType(statusCode), config.MonitorRegion) - fmt.Printf("[QUOTE-API][%s][paraswap][%s] %s | Latency: %.0fms | Status: %d\n", - timestamp, chain.Name, getStatusEmoji(statusCode), latencyMs, statusCode) - } else { - RecordQuoteAPILatency("paraswap", chain.Name, latencyMs, statusCode, config.MonitorRegion) - fmt.Printf("[QUOTE-API][%s][paraswap][%s] %s | Latency: %.0fms | Status: %d\n", - timestamp, chain.Name, getStatusEmoji(statusCode), latencyMs, statusCode) - } - - // Li.Fi (FREE) - latencyMs, statusCode, err = callLifiQuoteAPI(chain) - if err != nil || statusCode >= 400 { - RecordQuoteAPIError("lifi", chain.Name, getErrorType(statusCode), config.MonitorRegion) - fmt.Printf("[QUOTE-API][%s][lifi][%s] %s | Latency: %.0fms | Status: %d\n", - timestamp, chain.Name, getStatusEmoji(statusCode), latencyMs, statusCode) - } else { - RecordQuoteAPILatency("lifi", chain.Name, latencyMs, statusCode, config.MonitorRegion) - fmt.Printf("[QUOTE-API][%s][lifi][%s] %s | Latency: %.0fms | Status: %d\n", - timestamp, chain.Name, getStatusEmoji(statusCode), latencyMs, statusCode) - } - - // KyberSwap (FREE) - latencyMs, statusCode, err = callKyberSwapQuoteAPI(chain) - if err != nil || statusCode >= 400 { - RecordQuoteAPIError("kyberswap", chain.Name, getErrorType(statusCode), config.MonitorRegion) - fmt.Printf("[QUOTE-API][%s][kyberswap][%s] %s | Latency: %.0fms | Status: %d\n", - timestamp, chain.Name, getStatusEmoji(statusCode), latencyMs, statusCode) - } else { - RecordQuoteAPILatency("kyberswap", chain.Name, latencyMs, statusCode, config.MonitorRegion) - fmt.Printf("[QUOTE-API][%s][kyberswap][%s] %s | Latency: %.0fms | Status: %d\n", - timestamp, chain.Name, getStatusEmoji(statusCode), latencyMs, statusCode) - } - } - - // Jupiter (Solana) - Requires API key, skip if not available - // TODO: Add JUPITER_API_KEY to config if needed - // latencyMs, statusCode, err := callJupiterQuoteAPI("") - // ... - - fmt.Printf("[QUOTE-API][%s] === Quote API checks completed ===\n\n", timestamp) -} - -func getErrorType(statusCode int) string { - if statusCode >= 500 { - return "server_error" - } else if statusCode >= 400 { - return "client_error" - } else if statusCode == 0 { - return "timeout_error" - } - return "request_error" -} - -func getStatusEmoji(statusCode int) string { - if statusCode >= 400 { - return "✗" - } else if statusCode >= 300 { - return "⚠" - } - return "✓" -} - -// runQuoteAPIMonitor starts the quote API latency monitoring -func runQuoteAPIMonitor(config *Config, stopChan <-chan struct{}) { - fmt.Println("Starting Quote API Latency Monitor...") - fmt.Println(" Comparing: Mobula, Jupiter, OpenOcean, ParaSwap, Li.Fi, KyberSwap") - fmt.Println(" Mobula: Solana + Base") - fmt.Println(" Jupiter: Solana") - fmt.Println(" Others: Ethereum, Base, BNB") - fmt.Println(" Test: 100 USDC → Native token quote") - fmt.Println(" Interval: 30 seconds") - fmt.Println() - - // Create ticker for 30 second intervals - ticker := time.NewTicker(30 * time.Second) - defer ticker.Stop() - - // Run once immediately - performQuoteAPIChecks(config) - - // Then run every 30 seconds - for { - select { - case <-stopChan: - fmt.Println("Quote API monitor stopped") - return - case <-ticker.C: - performQuoteAPIChecks(config) - } - } -} - -// Helper to pretty print JSON for debugging -func prettyPrintJSON(data []byte) { - var prettyJSON map[string]interface{} - if err := json.Unmarshal(data, &prettyJSON); err == nil { - formatted, _ := json.MarshalIndent(prettyJSON, "", " ") - fmt.Printf("%s\n", formatted) - } -} diff --git a/harnesses/metadata-coverage/cmd/script/scrape_session.go b/harnesses/metadata-coverage/cmd/script/scrape_session.go deleted file mode 100644 index 4607912c..00000000 --- a/harnesses/metadata-coverage/cmd/script/scrape_session.go +++ /dev/null @@ -1,61 +0,0 @@ -package main - -import ( - "context" - "fmt" - "time" - - "github.com/chromedp/cdproto/network" - "github.com/chromedp/chromedp" -) - -func ScrapeDefinedSessionCookie() (string, error) { - // Create Chrome context with headless mode - opts := append(chromedp.DefaultExecAllocatorOptions[:], - chromedp.Flag("headless", true), - chromedp.Flag("disable-gpu", true), - chromedp.Flag("no-sandbox", true), - chromedp.Flag("disable-dev-shm-usage", true), - chromedp.UserAgent("Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"), - ) - - allocCtx, cancel := chromedp.NewExecAllocator(context.Background(), opts...) - defer cancel() - - ctx, cancel := chromedp.NewContext(allocCtx) - defer cancel() - - ctx, cancel = context.WithTimeout(ctx, 30*time.Second) - defer cancel() - - var sessionCookie string - - err := chromedp.Run(ctx, - chromedp.Navigate("https://www.defined.fi/"), - chromedp.WaitVisible(`body`, chromedp.ByQuery), - chromedp.Sleep(5*time.Second), - - chromedp.ActionFunc(func(ctx context.Context) error { - cookieParams, err := network.GetCookies().Do(ctx) - if err != nil { - return fmt.Errorf("failed to get cookies: %w", err) - } - - for _, cookie := range cookieParams { - if cookie.Name == "session" { - sessionCookie = cookie.Value - return nil - } - } - - return fmt.Errorf("session cookie not found") - }), - ) - - if err != nil { - return "", fmt.Errorf("failed to scrape session cookie: %w", err) - } - - return sessionCookie, nil -} - diff --git a/harnesses/metadata-coverage/go.mod b/harnesses/metadata-coverage/go.mod deleted file mode 100644 index ed17b543..00000000 --- a/harnesses/metadata-coverage/go.mod +++ /dev/null @@ -1,28 +0,0 @@ -module mobula_latency_competitor - -go 1.24.4 - -require ( - github.com/chromedp/cdproto v0.0.0-20250724212937-08a3db8b4327 - github.com/chromedp/chromedp v0.14.2 - github.com/gorilla/websocket v1.5.3 - github.com/prometheus/client_golang v1.23.2 -) - -require ( - github.com/beorn7/perks v1.0.1 // indirect - github.com/cespare/xxhash/v2 v2.3.0 // indirect - github.com/chromedp/sysutil v1.1.0 // indirect - github.com/go-json-experiment/json v0.0.0-20250725192818-e39067aee2d2 // indirect - github.com/gobwas/httphead v0.1.0 // indirect - github.com/gobwas/pool v0.2.1 // indirect - github.com/gobwas/ws v1.4.0 // indirect - github.com/kr/text v0.2.0 // indirect - github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect - github.com/prometheus/client_model v0.6.2 // indirect - github.com/prometheus/common v0.66.1 // indirect - github.com/prometheus/procfs v0.16.1 // indirect - go.yaml.in/yaml/v2 v2.4.2 // indirect - golang.org/x/sys v0.35.0 // indirect - google.golang.org/protobuf v1.36.8 // indirect -) diff --git a/harnesses/metadata-coverage/go.sum b/harnesses/metadata-coverage/go.sum deleted file mode 100644 index f594bc89..00000000 --- a/harnesses/metadata-coverage/go.sum +++ /dev/null @@ -1,67 +0,0 @@ -github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM= -github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw= -github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs= -github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs= -github.com/chromedp/cdproto v0.0.0-20250724212937-08a3db8b4327 h1:UQ4AU+BGti3Sy/aLU8KVseYKNALcX9UXY6DfpwQ6J8E= -github.com/chromedp/cdproto v0.0.0-20250724212937-08a3db8b4327/go.mod h1:NItd7aLkcfOA/dcMXvl8p1u+lQqioRMq/SqDp71Pb/k= -github.com/chromedp/chromedp v0.14.2 h1:r3b/WtwM50RsBZHMUm9fsNhhzRStTHrKdr2zmwbZSzM= -github.com/chromedp/chromedp v0.14.2/go.mod h1:rHzAv60xDE7VNy/MYtTUrYreSc0ujt2O1/C3bzctYBo= -github.com/chromedp/sysutil v1.1.0 h1:PUFNv5EcprjqXZD9nJb9b/c9ibAbxiYo4exNWZyipwM= -github.com/chromedp/sysutil v1.1.0/go.mod h1:WiThHUdltqCNKGc4gaU50XgYjwjYIhKWoHGPTUfWTJ8= -github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E= -github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= -github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= -github.com/go-json-experiment/json v0.0.0-20250725192818-e39067aee2d2 h1:iizUGZ9pEquQS5jTGkh4AqeeHCMbfbjeb0zMt0aEFzs= -github.com/go-json-experiment/json v0.0.0-20250725192818-e39067aee2d2/go.mod h1:TiCD2a1pcmjd7YnhGH0f/zKNcCD06B029pHhzV23c2M= -github.com/gobwas/httphead v0.1.0 h1:exrUm0f4YX0L7EBwZHuCF4GDp8aJfVeBrlLQrs6NqWU= -github.com/gobwas/httphead v0.1.0/go.mod h1:O/RXo79gxV8G+RqlR/otEwx4Q36zl9rqC5u12GKvMCM= -github.com/gobwas/pool v0.2.1 h1:xfeeEhW7pwmX8nuLVlqbzVc7udMDrwetjEv+TZIz1og= -github.com/gobwas/pool v0.2.1/go.mod h1:q8bcK0KcYlCgd9e7WYLm9LpyS+YeLd8JVDW6WezmKEw= -github.com/gobwas/ws v1.4.0 h1:CTaoG1tojrh4ucGPcoJFiAQUAsEWekEWvLy7GsVNqGs= -github.com/gobwas/ws v1.4.0/go.mod h1:G3gNqMNtPppf5XUz7O4shetPpcZ1VJ7zt18dlUeakrc= -github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8= -github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU= -github.com/gorilla/websocket v1.5.3 h1:saDtZ6Pbx/0u+bgYQ3q96pZgCzfhKXGPqt7kZ72aNNg= -github.com/gorilla/websocket v1.5.3/go.mod h1:YR8l580nyteQvAITg2hZ9XVh4b55+EU/adAjf1fMHhE= -github.com/klauspost/compress v1.18.0 h1:c/Cqfb0r+Yi+JtIEq73FWXVkRonBlf0CRNYc8Zttxdo= -github.com/klauspost/compress v1.18.0/go.mod h1:2Pp+KzxcywXVXMr50+X0Q/Lsb43OQHYWRCY2AiWywWQ= -github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE= -github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk= -github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY= -github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE= -github.com/kylelemons/godebug v1.1.0 h1:RPNrshWIDI6G2gRW9EHilWtl7Z6Sb1BR0xunSBf0SNc= -github.com/kylelemons/godebug v1.1.0/go.mod h1:9/0rRGxNHcop5bhtWyNeEfOS8JIWk580+fNqagV/RAw= -github.com/ledongthuc/pdf v0.0.0-20220302134840-0c2507a12d80 h1:6Yzfa6GP0rIo/kULo2bwGEkFvCePZ3qHDDTC3/J9Swo= -github.com/ledongthuc/pdf v0.0.0-20220302134840-0c2507a12d80/go.mod h1:imJHygn/1yfhB7XSJJKlFZKl/J+dCPAknuiaGOshXAs= -github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq1c1nUAm88MOHcQC9l5mIlSMApZMrHA= -github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ= -github.com/orisano/pixelmatch v0.0.0-20220722002657-fb0b55479cde h1:x0TT0RDC7UhAVbbWWBzr41ElhJx5tXPWkIHA2HWPRuw= -github.com/orisano/pixelmatch v0.0.0-20220722002657-fb0b55479cde/go.mod h1:nZgzbfBr3hhjoZnS66nKrHmduYNpc34ny7RK4z5/HM0= -github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= -github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= -github.com/prometheus/client_golang v1.23.2 h1:Je96obch5RDVy3FDMndoUsjAhG5Edi49h0RJWRi/o0o= -github.com/prometheus/client_golang v1.23.2/go.mod h1:Tb1a6LWHB3/SPIzCoaDXI4I8UHKeFTEQ1YCr+0Gyqmg= -github.com/prometheus/client_model v0.6.2 h1:oBsgwpGs7iVziMvrGhE53c/GrLUsZdHnqNwqPLxwZyk= -github.com/prometheus/client_model v0.6.2/go.mod h1:y3m2F6Gdpfy6Ut/GBsUqTWZqCUvMVzSfMLjcu6wAwpE= -github.com/prometheus/common v0.66.1 h1:h5E0h5/Y8niHc5DlaLlWLArTQI7tMrsfQjHV+d9ZoGs= -github.com/prometheus/common v0.66.1/go.mod h1:gcaUsgf3KfRSwHY4dIMXLPV0K/Wg1oZ8+SbZk/HH/dA= -github.com/prometheus/procfs v0.16.1 h1:hZ15bTNuirocR6u0JZ6BAHHmwS1p8B4P6MRqxtzMyRg= -github.com/prometheus/procfs v0.16.1/go.mod h1:teAbpZRB1iIAJYREa1LsoWUXykVXA1KlTmWl8x/U+Is= -github.com/rogpeppe/go-internal v1.10.0 h1:TMyTOH3F/DB16zRVcYyreMH6GnZZrwQVAoYjRBZyWFQ= -github.com/rogpeppe/go-internal v1.10.0/go.mod h1:UQnix2H7Ngw/k4C5ijL5+65zddjncjaFoBhdsK/akog= -github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U= -github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U= -go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto= -go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE= -go.yaml.in/yaml/v2 v2.4.2 h1:DzmwEr2rDGHl7lsFgAHxmNz/1NlQ7xLIrlN2h5d1eGI= -go.yaml.in/yaml/v2 v2.4.2/go.mod h1:081UH+NErpNdqlCXm3TtEran0rJZGxAYx9hb/ELlsPU= -golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.35.0 h1:vz1N37gP5bs89s7He8XuIYXpyY0+QlsKmzipCbUtyxI= -golang.org/x/sys v0.35.0/go.mod h1:BJP2sWEmIv4KK5OTEluFJCKSidICx8ciO85XgH3Ak8k= -google.golang.org/protobuf v1.36.8 h1:xHScyCOEuuwZEc6UtSOvPbAT4zRh0xcNRYekJwfqyMc= -google.golang.org/protobuf v1.36.8/go.mod h1:fuxRtAxBytpl4zzqUh6/eyUujkJdNiuEkXntxiD/uRU= -gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= -gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk= -gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q= -gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= -gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= diff --git a/infrastructure/prometheus/prometheus.yml b/infrastructure/prometheus/prometheus.yml index 3af3655b..cf1600a1 100644 --- a/infrastructure/prometheus/prometheus.yml +++ b/infrastructure/prometheus/prometheus.yml @@ -7,15 +7,12 @@ global: # OpenChainBench is a federation: each harness is hosted by whoever wrote it # (Mobula, an independent contributor, a provider that wants to be benchmarked). # This single shared Prometheus scrapes every harness's public /metrics -# endpoint and aggregates everything into one queryable instance, the URL +# endpoint and aggregates everything into one queryable instance — the URL # referenced by every YAML spec's `prom_url`. # -# One harness per benchmark. Adding a new bench means one new harness Go -# project, one new Railway service, one new scrape job below. -# # To onboard a new harness: # 1. The contributor exposes /metrics on a publicly reachable URL -# (Railway, Fly, Cloud Run, a VPS, anywhere with HTTPS). +# (Railway, Fly, Cloud Run, a VPS — anywhere with HTTPS). # 2. Append a `- job_name: ` block below pointing at that URL. # 3. Open a PR. Once merged, redeploy the prometheus service so it picks # up the new target (or hit the /-/reload endpoint). @@ -24,8 +21,8 @@ global: # use it to scope queries. scrape_configs: - # ─ № 001 · aggregator-head-lag ─────────────────────────────────────── - # Hosted by Mobula on Railway. WebSocket monitor + canonical-tip checks. + # ─ Aggregator head lag ──────────────────────────────────────────────── + # Currently hosted by Mobula on Railway. Replace the URL if you fork. - job_name: aggregator-head-lag metrics_path: /metrics scheme: https @@ -36,41 +33,16 @@ scrape_configs: benchmark: aggregator-head-lag host: mobula - # ─ № 002 · bridge-quote-latency ────────────────────────────────────── - # Hosted by Mobula on Railway. Bridge quote loop, latency-focused. - - job_name: bridge-quote-latency - metrics_path: /metrics - scheme: https - static_configs: - - targets: - - bridge-quote-latency-production.up.railway.app - labels: - benchmark: bridge-quote-latency - host: mobula - - # ─ № 003 · bridge-fee ──────────────────────────────────────────────── - # Hosted by Mobula on Railway. Bridge cost-percent measurement. - - job_name: bridge-fee - metrics_path: /metrics - scheme: https - static_configs: - - targets: - - bridge-fee-production.up.railway.app - labels: - benchmark: bridge-fee - host: mobula - - # ─ № 004 · metadata-coverage ───────────────────────────────────────── - # Hosted by Mobula on Railway. Pulse V2 feeder + per-aggregator metadata - # field checks (logo / description / twitter / website). - - job_name: metadata-coverage + # ─ Bridge quote latency + cost ──────────────────────────────────────── + # Currently hosted by Mobula on Railway. Replace the URL if you fork. + - job_name: bridge-monitor metrics_path: /metrics scheme: https static_configs: - targets: - - metadata-coverage-production.up.railway.app + - bridge-monitor-production.up.railway.app labels: - benchmark: metadata-coverage + benchmark: bridge-monitor host: mobula # ─ Self-scrape ────────────────────────────────────────────────────────