fix(bench): sample-size PromQL + leader reliability floor - #1242
Merged
Conversation
* fix(bench): sample-size PromQL + leader reliability floor Two bench-correctness fixes surfaced by an audit of the oracle and gas surfaces we plan to cite in outreach. Bench 025 (oracle-deviation): - sample_size was 'count(count by (source) (ocb_oracle_price))' which returns the count of distinct sources (3 or 4), not the count of samples over the window. The citable API therefore reported sampleSize=3-4 on every pair, at odds with the spec's 'polled every 30s' claim. Rewrites the query to 'sum(count_over_time(ocb_oracle_price[24h]))' across all 10 pair entries so the number reflects actual poll volume. Leader ranking: - citationCandidates() now excludes providers with a measured success rate below 50 percent from the leader pool. Before this fix, gas-estimation showed Owlracle as #1 with successRate=6.48 and a p50 gap of 0.001 gwei: technically accurate on the rare successful poll but useless as a leader claim. Falls back to the full live pool when every provider is under the floor so a totally degraded bench still surfaces a best-of-bad-options leader rather than vanishing. Freshness benches without a success query default to 100 percent in the loader so the guard is inert there. Adds src/lib/citation.test.ts with 5 tests covering the Owlracle scenario, higher-is-better ranking, and the degraded-pool fallback. * fix(bench): propagate reliability floor to rankings + template + hub Follow-up on the previous commit. Review flagged that filtering the leader without filtering the ranking arrays leaves self-contradictory JSON: /api/stat can report leader=Etherscan while rankings[0]=Owlracle on the same document. Exports citationCandidates and adds rankedCandidates helper that returns the same eligible pool sorted by the bench's higher_is_better direction. leader() and fieldValue() now share it. Wires the shared helper into every surface that emits a rankings-like list or a best-of claim: - src/app/api/stat/[slug]: rankings array now sorted from rankedCandidates so rankings[0].slug always equals leader.slug on the same JSON. - src/app/api/llm-context: the Rankings numbered list under each bench matches the Headline sentence above it. - src/app/api/mcp/[transport]: get_benchmark tool response and the openchainbench://benchmark/{slug} resource both share the pool. - src/lib/bench-template: {{best_name}} / {{best_p50}} / {{worst_name}} tokens in bench copy resolve against the same pool so a bench body never crowns a provider its own headline excludes. Per-slug tokens {{p50:some-slug}} still resolve against the raw live pool so unknown tokens still fall through. - src/lib/providers: rankProviders (drives /products hub wins count and per-chain leadership chips) now runs on the reliable pool. Adds a consistency test locking that rankedCandidates[0].slug always matches leader().slug on the same bench. Total: 6 tests in citation.test.ts, 95/95 pass across src/. Uses (r.successRate ?? 100) for the numeric guard to future-proof against ProviderResult shapes that skip the field. Load path still defaults to 100 (materialize/load.ts:748) so this is inert today, but the extra ?? removes the fragility. /api/compare/[a]/[b] is intentionally left unfiltered: the user picks the two providers explicitly, so the head-to-head is a requested comparison rather than a leader claim. Applying the floor there would drop rows the user explicitly asked to see. * fix(hub-card): honor citation leader in the hub grid headline Follow-up review flagged that benchmark-card.tsx re-sorted b.results locally and picked results[0], so a bench where citation demotes the raw-p50 winner (e.g. Owlracle on gas-estimation) would show the demoted provider on the hub grid while the bench page names the correct leader. Two documents contradicting each other in one click. Adds leaderSlug: string | null to BenchmarkCardData and populates it at the projection boundary via leader(b), which already applies the reliability + insufficient-sample filter. The card picks the leader by matching that slug in the projected results, and falls back to the raw best when the projection returned no leader (draft, insufficient, or all providers filtered out) so a card still renders a headline value in every state. Kept the local sort so the fallback path and any downstream chip using sorted[0].name keeps working. Test suite unchanged: 95/95 pass across src/. No test needed since this is a projection-boundary wire-up. --------- Co-authored-by: Florent Tapponnier <contact@mobula.io>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cherry-pick to main of #1175 (which landed on dev but was never synced to main).
The bug
Live prod
GET /api/stat/rpc-capabilities?chain=bnbreturnsleader: {name: 'dRPC', value: 17, successRate: 25.4%}— dRPC fails 3 out of 4 calls on BNB but is promoted as leader anyway. The methodology says 'read rows below 50% success as degraded' but the leader field ignored it.The fix
LEADER_MIN_SUCCESS_PCT = 50in src/lib/citation.tscitationCandidates()drops rows below 50% success (with escape hatch: if ALL are degraded, fall back to full pool)leader()andrankedCandidates()now share the filtered pool →rankings[0]stays consistent withleaderVerification
pnpm typecheck✅bun test src/lib/citation.test.ts→ 6/6 passGET /api/stat/rpc-capabilities?chain=bnbwill return Binance as leader (100% success, 74ms)