diff --git a/benchmarks/solana-tx-landing-latency.yml b/benchmarks/solana-tx-landing-latency.yml new file mode 100644 index 00000000..9bb14231 --- /dev/null +++ b/benchmarks/solana-tx-landing-latency.yml @@ -0,0 +1,221 @@ +# OpenChainBench. Bench № 027 + +slug: solana-tx-landing-latency +number: "027" +title: Solana transaction landing latency +seo_title: "Solana tx landing latency 2026: Jito vs Helius vs Nozomi vs Astralane vs Mobula time-to-land in slots" +seo_description: "Live active benchmark of Solana transaction landing services. Slot-level time-to-land (canonical on-chain measurement) plus wall-clock milliseconds. Identical signed mainnet probes submitted to each service (Jito, Helius Sender, Astralane, Nozomi, Mobula) from us-east every hour. Methodology pre-registered." +subtitle: How fast does each landing service get a signed mainnet tx confirmed. Slot delta = number of Solana slots between submit and confirmed (1 slot ≈ 400 ms). Active probing, identical signed txs every hour from us-east. +category: Trading +status: live +metric: p50 slot delta to confirmed (7-day window) +unit: slots +higher_is_better: false + +disclaimer: | + Six caveats. (1) us-east only, sgp / eu-west arrive in V2. (2) One pre-registered tip per service. (3) Synthetic payload (1-lamport + memo); real swaps may land differently. (4) Helius / Astralane / Nozomi fan out to Jito internally; Jito control probe runs each cycle. (5) Confirmation = `confirmed`. (6) Slot delta is canonical; ms is derived (≈ slot_delta × 400 ms + RTT). Pair with /benchmarks/solana-tx-landing. + +seo_intro: | + This benchmark answers the only question that matters to a + Solana trader picking a landing service. how many slots does + your signed mainnet transaction take to reach the confirmed + state on chain. Every hour from a us-east probe, the harness + submits an identical signed tx through each of 5 services in + parallel, captures the submit slot before send and the land + slot from the signatureSubscribe WebSocket notification at + commitment=confirmed, and increments per-service Prometheus + histograms. Headline numbers shown are p50 and p99 slot delta + over a rolling 7-day window. Wall-clock milliseconds are + published alongside for intuition (one Solana slot is ~400 ms, + so a p50 of 1 slot is ~400 ms wall-clock plus submission RTT) + but slot delta is the canonical, sponsor-proof on-chain + measurement. + Why slot delta is the right metric. Solana confirmation is a + slot-level event. when a slot reaches supermajority vote, every + transaction in it becomes confirmed simultaneously. Wall-clock + ms conflates HTTP submission time, our RPC's polling lag, and + network RTT to the public WebSocket - all of which are + measurement artifacts unrelated to the landing service's actual + routing quality. Slot delta is what the chain itself records. + Coverage. 5 services probed in V0-Lean. Jito Block Engine (the + control / baseline because Helius, Astralane, Nozomi all + internally route some flow through it). Helius Sender in + `swqos_only=true` mode (isolates the Helius own-path from the + Jito leg). Astralane Iris (tip-refund mechanism). Nozomi by + Temporal Labs (premium tier, hard 1M lamport tip floor). + 0slot.trade (premium tier). NextBlock, bloXroute and + SolanaVibeStation arrive in the next tier (V1) once the first + sponsors land. Companion bench. /benchmarks/solana-tx-landing + measures market share via on-chain tip-wallet attribution - + who carries the flow today, regardless of speed. + +abstract: | + We probe 5 Solana transaction landing services from a single + Railway us-east region, once per hour, by submitting an + identical signed mainnet transaction to each. The payload is + the minimal valid Solana tx, compute-budget instructions + (50k CU limit, 50k micro-lamport/CU price), a 1-lamport + self-transfer, the per-service tip transfer to the service's + documented tip wallet, and an OCB-prefixed memo for forensic + traceability. All five services are submitted in parallel + goroutines within a single cycle so they sample the same chain + congestion window. The headline measurement is slot delta, + land_slot minus submit_slot, captured from the + signatureSubscribe WebSocket notification's context.slot field + at commitment=confirmed. Wall-clock ms is reported alongside + but is a derived approximation, slot_delta × ~400 ms plus + submission RTT and goroutine startup variance. A 60 s no- + confirmation deadline classifies the probe as + dropped{reason=timeout}; structured RPC errors classify as + invalid; transport failures as network_error; HTTP 419 / 429 / + "rate limit" errors classify as rate_limited (a separate label + so quota issues don't bias the bench against the throttled + service). Cost. ~$159/mo at SOL=$86, 86 % of which goes to the + four ≥1M-lamport-floor services (Nozomi, 0slot, bloXroute, + NextBlock, only two of these in V0-Lean). Sponsor SOL credits + covering a service's own probes are explicitly allowed per the + sponsor-proof framework. Limitations. (a) Single us-east + region, sgp / eu-west arrive in V2 once sponsors fund + geographic-edge story. (b) 1-hour cadence, 168 probes per + service per 7-day window, enough for stable p50 / p99 over the + publication window, not enough for intra-hour resolution + (V0.5 / V1 upgrade if needed). (c) Fan-out, Helius probed in + `swqos_only` mode only in V0-Lean to keep wire shape simple; + dual-mode arrives in v1.0.1 methodology PR. + +methodology: + - "Source endpoints (us-east Railway, base64 JSON-RPC sendTransaction unless noted). Jito `ny.mainnet.block-engine.jito.wtf/api/v1/transactions`. Helius Sender `ewr-sender.helius-rpc.com/fast?swqos_only=true` (skipPreflight + maxRetries=0). Nozomi `http://edge.nozomi.temporal.xyz/api/sendBatch?c=` (binary `[u16_BE_len][tx_bytes]`, HTTP per Temporal Labs). Astralane `ny.gateway.astralane.io/iris?api-key=` (3-elem params, mevProtect). 0slot `ny.0slot.trade?api-key=`." + - "Probe payload. 5 instructions in this exact order: SetComputeUnitLimit(50,000) + SetComputeUnitPrice(50,000 micro-lamports/CU) + SystemProgram.Transfer(payer→payer, 1 lamport) + SystemProgram.Transfer(payer→service tip wallet, floor lamports) + Memo(`ocb---`). cycle_id is an 8-byte random hex shared across the five parallel probes of one cycle, so the on-chain memos correlate." + - "Tip floors (pre-registered, methodology PR + 14-day window to change). Jito 10,000 lamports. Helius Sender 10,000. Astralane 500,000 net of refunds. Nozomi 1,000,000. 0slot 1,000,000." + - "Submission flow. One getLatestBlockhash(processed) shared across all five probes. One getSlot(processed) as submit_slot. For each service we subscribe to the signature via signatureSubscribe on the public WS BEFORE submission (otherwise a fast confirm could fire before we listen). Probes then fire in parallel goroutines, sign, POST. We block on the signatureNotification at commitment=confirmed; context.slot is land_slot; slot_delta = land_slot - submit_slot." + - "Why slot delta is canonical. Solana confirmation is slot-level. when a slot reaches supermajority, every tx in it becomes confirmed simultaneously. The WS pushes notifications for all subscribed sigs in that slot at the same instant. So sub-400 ms wallclock diffs between services in the same slot are artifacts (goroutine startup, RTT), not routing quality. slot_delta is what the chain records, what to cite in audits." + - "Wall-clock ms is a derived approximation. ms ≈ slot_delta × ~400 ms + HTTP submission RTT + variance. We publish it for intuition because traders think in seconds, not slots, but it should not be the sole metric in a sponsor pitch or audit. If a service argues 'your ms numbers are biased by your RTT', the answer is the slot delta column, which is RTT-independent." + - "Drop classification. timeout = no confirmation within 60 s. invalid = RPC error, on-chain Err, or BlockhashNotFound. network_error = transport-level (timeout, DNS, EOF, connection refused). rate_limited = HTTP 419 / 429 / 'rate limit' / 'too many requests'. landing_rate is published as success / (success + timeout), rate_limited and network_error are excluded so quota / transport issues don't bias the bench against a throttled service." + - "Jito control probe. Helius (default), Astralane, Nozomi route a portion of flow through Jito internally, conflating own-path vs Jito-caught-it. Jito is in the V0-Lean set so its control fires in the same cycle. Same slot_delta as Jito = no measurable own-path value. Suspect ahead by 1+ slot = real routing advantage." + - "Reproducibility. The full harness source is at github.com/MobulaFi/mobula-monorepo/tree/main/miniapps/solana-tx-landing. Anyone with a funded Solana keypair (~1 SOL) can clone, set SOLANA_PROBE_KEYPAIR_BASE58, run the binary, and reproduce these metrics. The bench does not rely on any private or internal service for measurement, the only RPC dependency is the public `api.mainnet-beta.solana.com` HTTP + WebSocket endpoints." + - "Methodology v1.3 pre-registered at github.com/ChainBench/OpenChainBench/blob/main/docs/methodology/solana-tx-landing-active.md. Any change (tip floor, probe payload, cadence, region, metric definitions) ships as a public PR with a 14-day comment window. Major version bumps run a 30-day shadow period publishing old and new metrics in parallel." + +findings: + - "{{best_name}} leads the V0-Lean probe set at p50 = {{best_p50}} slot delta over the rolling 7-day window. Lower = fewer Solana slots between submission and confirmation. The gap between fastest and slowest is the operational signal, every service claims '99 %+ landing rate' in marketing copy, but the chain doesn't lie about which slot included your tx. A 1-slot difference is ~400 ms, enough for a MEV bot to front-run a competitor." + - "{{name:jito}} is the baseline / control. Helius (default mode), Astralane, and Nozomi all internally fan out to Jito, so the Jito p50 is the floor any premium service must beat. Same slot_delta as Jito on a given cycle = the service is essentially using Jito as its inclusion path. {{name:jito}} sits at p50 = {{p50:jito}} slot delta." + - "{{name:helius-sender}} in `swqos_only` mode isolates Helius's own routing path from the Jito leg. p50 = {{p50:helius-sender}} slot delta. A v1.0.1 methodology update will publish Helius default mode (with Jito fan-out) side-by-side for direct comparison." + - "{{name:nozomi}} premium pricing (1M lamport hard floor, ~10 × Jito's competitive level) only makes economic sense if the slot_delta advantage is meaningful. p50 = {{p50:nozomi}} slot delta. The gap vs Jito quantifies whether the tip premium buys real slot priority." + - "{{worst_name}} trails at p50 = {{worst_p50}} slot delta. The worst slot delta in the V0-Lean set is not necessarily a bad service, it may be a service whose strength is in dimensions this bench doesn't measure (anti-MEV protection, durable nonce, fee-refund mechanics). Latency is one variable, not the whole product." + +faq: + - q: "Why is slot delta the headline metric instead of wall-clock latency?" + a: "Solana confirmation is a slot-level event. when a slot reaches supermajority vote (~2/3 of stake), every transaction in that slot becomes confirmed simultaneously. The WebSocket pushes notifications for all subscribed signatures in that slot at the same instant. So if 3 services delivered txs that all landed in the same slot, our wallclock measurement records the same time for all 3, the only differentiation is whether the next service's tx landed in slot N or N+1. slot_delta captures that directly. Wall-clock ms is derived (slot_delta × ~400 ms + RTT + variance) and conflates routing quality with measurement artifacts like HTTP submission speed and our public RPC's network latency. We publish wall-clock ms because traders think in seconds, but slot_delta is what you should cite in an audit or methodology dispute. It's RTT-independent and reads directly from the chain." + - q: "What does '1 slot' actually mean in time?" + a: "Solana slots are ~400 ms in practice (~625 ms target with leader skips and forks averaging it down). A p50 slot_delta of 1 means your tx typically lands in the slot immediately following your submission, ~400 ms after sendTransaction return. p50 of 2 means typically one slot later, ~800 ms. The gap between p50 = 1 and p50 = 2 is the operational signal, a service that consistently lands 1 slot earlier than its competitors is ~400 ms ahead, which is the difference between catching an arbitrage and missing it." + - q: "Why an active bench when /benchmarks/solana-tx-landing already exists?" + a: "/benchmarks/solana-tx-landing is observational, it watches the chain and counts who carries the flow. It cannot answer 'how fast does my tx land if I send it now', because it doesn't send anything. This bench (active probing) answers that, at the cost of running 24 / 7 with real SOL ($159 / month at the V0-Lean cadence). The two benches answer different product questions. Read both." + - q: "Why only 5 services, not the 8 you measure observationally?" + a: "NextBlock, bloXroute Trader, and SolanaVibeStation all require paid plans or sales-call onboarding before they issue an API key. We're shipping V0-Lean today with the 5 services that have a clear self-serve or contact-based path. The other 3 will be added as the bench scales. The observational bench at /benchmarks/solana-tx-landing already covers all 8 because it doesn't need API keys." + - q: "Why us-east only?" + a: "V0-Lean. us-east is the de-facto Solana baseline (Jito, NextBlock, bloXroute, Helius all anchor their best-connected POPs there) and is where most Solana bots deploy by default. Adding eu-west and sgp triples the bench cost and answers a different question ('does the ranking change by geography?'), which is a planned V2 scope expansion." + - q: "What's the probe payload?" + a: "Five instructions in this exact order, locked by methodology §3. (1) SetComputeUnitLimit(50,000). (2) SetComputeUnitPrice(50,000 micro-lamports/CU), together a 2,500-lamport priority fee. (3) SystemProgram.Transfer of 1 lamport from the prober keypair to itself, the minimal valid state-touching tx. (4) SystemProgram.Transfer to the service's documented tip wallet at the pre-registered floor. (5) Memo program write with the cycle ID, service name, and probe mode. Total weight: ~600 bytes, well under the 1,232-byte tx limit." + - q: "How is fan-out handled?" + a: "Helius (default mode), Astralane, and Nozomi route a portion of flow through Jito internally. The Jito control probe, Jito is part of the V0-Lean probe set, fires in the same cycle as the suspect services with the same blockhash and a comparable tip. The slot_delta column tells you immediately whether a suspect service is adding value beyond a Jito wrapper. Same slot_delta as Jito = same inclusion slot = Jito caught it. Suspect ahead by 1+ slot = real own-path routing. Helius is additionally probed in `?swqos_only=true` mode to fully isolate its own routing path." + - q: "Can a service detect and prioritise our probes?" + a: "Yes, in principle. The memo prefix `ocb-` is deterministic and the keypair is constant per region. Anti-fingerprinting (memo randomisation, sub-account rotation, tip jitter within the floor band) ships in v1.0.2 methodology PR. We disclose this risk openly; the trade-off is that announcing the bench publicly to providers gives them a chance to fix real performance issues before we publish, which is a good outcome. We do NOT accept private deals to alter the probe surface for any specific service." + - q: "Why is sample_size on the dashboard ~168 per service?" + a: "V0-Lean cadence = 1 probe per service per hour from 1 region. 168 = 24 hours × 7 days. The 7-day publication window is the trade-off between statistical resolution (sample size grows with window) and freshness (shorter window reflects current chain conditions). At ~168 samples per cell, p50 is stable to within ±5 % and p99 to within ±15 %. Lower confidence intervals are unlocked at V0.5 cadence (1 / 10 min, ~$760 / mo) and above." + - q: "How is the confirmation observed?" + a: "Via `signatureSubscribe` on the public mainnet WebSocket (`wss://api.mainnet-beta.solana.com`). The subscription is registered BEFORE submission so a fast-confirming tx cannot complete before we are listening (otherwise we'd miss the notification and timeout spuriously). The RPC pushes the notification at the instant the commitment level is reached, so observation resolution is RTT-bounded (~30-50 ms us-east → mainnet-beta) and slot_delta is read directly from the notification's context.slot field. HTTP polling at 200 ms is an automatic fallback if the WebSocket fails to connect for a given cycle." + +source: https://github.com/ChainBench/OpenChainBench/tree/main/harnesses/solana-tx-landing + +prometheus: + window: 7d + expected_freshness_seconds: 7200 + +# Real metrics emitted by the active prober in solana-tx-landing harness: +# solana_landing_probe_success_total{service, mode, region} counter +# solana_landing_probe_dropped_total{service, mode, region, reason} counter +# solana_landing_probe_latency_ms{service, mode, region} gauge (set every cycle) +# solana_landing_probe_latency_slots{service, mode, region} gauge (set every cycle) +# solana_landing_probe_latency_slots_histogram{service, mode, region} histogram (debug) +# solana_landing_probe_latency_ms_histogram{service, mode, region} histogram (debug) +# solana_landing_probe_keypair_balance_sol{region} gauge +# solana_landing_probe_cycle_total{region} counter +# solana_landing_probe_enabled{region} gauge +# +# Headline metric (canonical) = slot_p50 / slot_p99 read from the gauge. +# Wall-clock ms is published alongside via the standard p50/p90/p99 fields +# for reader intuition but is derived (slot_delta × ~400 ms + RTT + variance). +# Mode label is `swqos_only` for helius-sender, `default` for the rest. +# +# Why quantile_over_time(gauge) instead of histogram_quantile(histogram)? +# At V0-Lean cadence (1 probe / hour) we have ~168 samples per cell over 7d. +# Histogram buckets {100, 250, 500, 1000, 2000, 5000, 10000, 30000, 60000} ms +# have ~3 buckets in the 1-5s zone where probes actually land, so +# histogram_quantile collapses to bucket midpoints (1500, 3500 ms) and the +# series looks flat. quantile_over_time on the gauge takes the real sample +# at the 50th percentile, which is the accurate published number. + +providers: + - slug: jito + name: Jito + tag: Baseline + control probe; atomic bundles + tip auction since 2022 + formula: "50th percentile over 7d of slot delta (land_slot − submit_slot) for hourly signed mainnet probes submitted to Jito's `ny.mainnet.block-engine.jito.wtf` from us-east." + queries: + p50: quantile_over_time(0.5, solana_landing_probe_latency_slots{service="jito",region="us-east"}[7d]) + p90: quantile_over_time(0.9, solana_landing_probe_latency_slots{service="jito",region="us-east"}[7d]) + p99: quantile_over_time(0.99, solana_landing_probe_latency_slots{service="jito",region="us-east"}[7d]) + mean: avg_over_time(solana_landing_probe_latency_slots{service="jito",region="us-east"}[7d]) + success: sum(rate(solana_landing_probe_success_total{service="jito",region="us-east"}[7d])) / (sum(rate(solana_landing_probe_success_total{service="jito",region="us-east"}[7d])) + sum(rate(solana_landing_probe_dropped_total{service="jito",region="us-east",reason="timeout"}[7d]))) + sample_size: sum(increase(solana_landing_probe_success_total{service="jito",region="us-east"}[7d])) + series: solana_landing_probe_latency_slots{service="jito",region="us-east"} + + - slug: helius-sender + name: Helius + tag: Isolated Helius own-path (no Jito fan-out); anycast + 7 POPs + formula: "50th percentile over 7d of slot delta (land_slot − submit_slot) for hourly signed probes submitted to Helius Sender in `swqos_only=true` mode from us-east, isolating its own-path." + queries: + p50: quantile_over_time(0.5, solana_landing_probe_latency_slots{service="helius-sender",mode="swqos_only",region="us-east"}[7d]) + p90: quantile_over_time(0.9, solana_landing_probe_latency_slots{service="helius-sender",mode="swqos_only",region="us-east"}[7d]) + p99: quantile_over_time(0.99, solana_landing_probe_latency_slots{service="helius-sender",mode="swqos_only",region="us-east"}[7d]) + mean: avg_over_time(solana_landing_probe_latency_slots{service="helius-sender",mode="swqos_only",region="us-east"}[7d]) + success: sum(rate(solana_landing_probe_success_total{service="helius-sender",mode="swqos_only",region="us-east"}[7d])) / (sum(rate(solana_landing_probe_success_total{service="helius-sender",mode="swqos_only",region="us-east"}[7d])) + sum(rate(solana_landing_probe_dropped_total{service="helius-sender",mode="swqos_only",region="us-east",reason="timeout"}[7d]))) + sample_size: sum(increase(solana_landing_probe_success_total{service="helius-sender",mode="swqos_only",region="us-east"}[7d])) + series: solana_landing_probe_latency_slots{service="helius-sender",mode="swqos_only",region="us-east"} + + - slug: astralane + name: Astralane + tag: Tip-refund mechanism, sendBundle / sendIdeal modes, FRA + NY POPs + formula: "50th percentile over 7d of slot delta (land_slot − submit_slot) for hourly signed probes submitted with a 500k-lamport net tip to Astralane Iris's NY gateway from us-east." + queries: + p50: quantile_over_time(0.5, solana_landing_probe_latency_slots{service="astralane",region="us-east"}[7d]) + p90: quantile_over_time(0.9, solana_landing_probe_latency_slots{service="astralane",region="us-east"}[7d]) + p99: quantile_over_time(0.99, solana_landing_probe_latency_slots{service="astralane",region="us-east"}[7d]) + mean: avg_over_time(solana_landing_probe_latency_slots{service="astralane",region="us-east"}[7d]) + success: sum(rate(solana_landing_probe_success_total{service="astralane",region="us-east"}[7d])) / (sum(rate(solana_landing_probe_success_total{service="astralane",region="us-east"}[7d])) + sum(rate(solana_landing_probe_dropped_total{service="astralane",region="us-east",reason="timeout"}[7d]))) + sample_size: sum(increase(solana_landing_probe_success_total{service="astralane",region="us-east"}[7d])) + series: solana_landing_probe_latency_slots{service="astralane",region="us-east"} + + - slug: nozomi + name: Nozomi + tag: Temporal Labs, direct-to-leader, premium 1M-lamport hard floor + formula: "50th percentile over 7d of slot delta (land_slot − submit_slot) for hourly signed probes submitted with a 1M-lamport tip to Nozomi's `edge.nozomi.temporal.xyz` from us-east." + queries: + p50: quantile_over_time(0.5, solana_landing_probe_latency_slots{service="nozomi",region="us-east"}[7d]) + p90: quantile_over_time(0.9, solana_landing_probe_latency_slots{service="nozomi",region="us-east"}[7d]) + p99: quantile_over_time(0.99, solana_landing_probe_latency_slots{service="nozomi",region="us-east"}[7d]) + mean: avg_over_time(solana_landing_probe_latency_slots{service="nozomi",region="us-east"}[7d]) + success: sum(rate(solana_landing_probe_success_total{service="nozomi",region="us-east"}[7d])) / (sum(rate(solana_landing_probe_success_total{service="nozomi",region="us-east"}[7d])) + sum(rate(solana_landing_probe_dropped_total{service="nozomi",region="us-east",reason="timeout"}[7d]))) + sample_size: sum(increase(solana_landing_probe_success_total{service="nozomi",region="us-east"}[7d])) + series: solana_landing_probe_latency_slots{service="nozomi",region="us-east"} + + - slug: mobula + name: Mobula + tag: Multi-RPC fan-out aggregator (relays via Jito / Nozomi / zeroslot) + formula: "50th percentile over 7d of slot delta (land_slot − submit_slot) for hourly probes submitted via Mobula's `api.mobula.io/api/2/swap/send` multi-RPC fan-out from us-east, using a Jito tip wallet." + queries: + p50: quantile_over_time(0.5, solana_landing_probe_latency_slots{service="mobula",region="us-east"}[7d]) + p90: quantile_over_time(0.9, solana_landing_probe_latency_slots{service="mobula",region="us-east"}[7d]) + p99: quantile_over_time(0.99, solana_landing_probe_latency_slots{service="mobula",region="us-east"}[7d]) + mean: avg_over_time(solana_landing_probe_latency_slots{service="mobula",region="us-east"}[7d]) + success: sum(rate(solana_landing_probe_success_total{service="mobula",region="us-east"}[7d])) / (sum(rate(solana_landing_probe_success_total{service="mobula",region="us-east"}[7d])) + sum(rate(solana_landing_probe_dropped_total{service="mobula",region="us-east",reason="timeout"}[7d]))) + sample_size: sum(increase(solana_landing_probe_success_total{service="mobula",region="us-east"}[7d])) + series: solana_landing_probe_latency_slots{service="mobula",region="us-east"}