feat(sample-health): per-provider data confidence + aggregate (cherry-pick A) - #732
Merged
Merged
Conversation
…ate (cherry-pick A from dev) First slice of the dev → main reconciliation. Pure additive: brings the sample-health classification system from dev so bench pages and citable APIs can honestly badge undersized samples instead of crowning a leader drawn from 3 data points. The motivation: bench pages used to rank providers from sample counts spanning 3 to tens of thousands without surfacing confidence level, while the methodology page advertised 'n >= 1000 per provider'. On low-cadence benches like perp-funding (3 series per venue by design) that methodology line read as broken — an E-E-A-T penalty waiting to happen with Google. A flat threshold mis-classifies legitimately low-cadence benches. Per-bench expected_n declared in the YAML lets each bench publish what 'full coverage' looks like; this PR computes health = sampleSize / expected_n per provider: - healthy ratio >= 0.5 - low 0.1 <= ratio < 0.5 - insufficient ratio < 0.1 What's wired in this PR: - src/lib/sample-health.ts (new): classifyHealth + aggregateConfidence - src/lib/sample-health.test.ts (new): 11 cases covering edges - src/lib/time-constants.ts (new): shared MS_PER_* constants - src/lib/spec-schema.ts: optional expected_n field (additive) - src/types/benchmark.ts: optional expectedN / dataConfidence / sampleHealth fields on Benchmark + ProviderResult (additive) - src/lib/materialize/load.ts: classifyHealth per result + aggregate the bench-wide confidence when spec declares expected_n - src/lib/spec.ts: overlayEditorial passes through expectedN so a YAML edit takes effect before the materialise worker re-snapshots - Cache keys bumped bench-unfiltered v13 → v14, all-benchmarks v17 → v18 (regenerate stored snapshots through the new path) What's deliberately NOT in this PR (follow-ups): - UI gating in ledger-table.tsx (Low sample pill, hide insufficient rows from ranking) — separate PR - citable / stat / openapi.json API surfacing — separate PR - citation.ts updates — separate PR - YAML expected_n declarations on individual benches — content follow-ups, one bench at a time Benches without expected_n behave exactly as before (the field is optional, the classifier returns null, no badge renders, no behavior change). Zero risk to existing prod surfaces.
Flotapponnier
added a commit
that referenced
this pull request
Jun 26, 2026
…emas (PR E) (#739) PR E in dev → main reconciliation. Minimal-change strategy: main was already ahead on spec.ts (chain-aliases reconciliation, expectedN overlay, quorum guard all already cherry-picked). Dev only brings 2 real additions: spec.ts changes (cosmetic): - Import MS_PER_MINUTE from @/lib/time-constants - Replace 30 * 60_000 → 30 * MS_PER_MINUTE (line 50) - Replace / 60000 → / MS_PER_MINUTE in log fmt (line 61) - Cache keys bumped bench-unfiltered v14 → v15, all-benchmarks v18 → v19 (regenerate stored snapshots through the new shape v5 below) - Everything else preserved: chain-aliases import, bestForChain re-export, overlayEditorial canonical-slug reconciliation, expectedN passthrough, AllBenchmarksDraftError, quorum guard. snapshot.ts changes (real upgrade): - Replace full file with dev's 441-line version - Was: 323 lines with z.any() loose validation - Now: strict Zod schemas for ProviderResult / ResultExtras / MetricPanel / CellRankEntry / StalenessMeta / RegionPoint - Adds AssertEqual compile-time guards (4 const checks) that fail the build if @/types/benchmark drifts from the Zod schemas - SnapshotPayload adopts expectedN + dataConfidence (completes PR #732 sample-health on the persistence side — sample-health fields now survive KV roundtrip) - snapshotFromBenchmark writes the new fields too - SCHEMA_VERSION bumped 3 → 5 (covers both branches' versions) Old v3 blobs in Upstash KV will safeParse → null → live Prom path takes over and rewrites under v5. Graceful degradation intact (verified in readSnapshot at line 373). - Removes 'as ProviderResult[]' casts (Zod now guarantees shape) Risk: LOW (per second-pass review by reviewer agent). - No external callers of snapshot.ts (only spec.ts imports it). - No external constructors of SnapshotPayload (only snapshotFromBenchmark builds it). - One-time cold render per (slug, variant) on first reader after deploy. With READ_FROM_STORE=1 the worker snapshots absorb it; without, it's a single Prom fan-out per bench. - No caller signature changes. Co-authored-by: Florent Tapponnier <contact@mobula.io>
Flotapponnier
added a commit
that referenced
this pull request
Jun 26, 2026
…cing (follow-up A) (#741) Completes PR #732 (sample-health system) by surfacing the computed dataConfidence/sampleHealth/expectedN to users + machine-readable consumers. Was dead-code on the data layer until this PR. UI (ledger-table.tsx): - Filter rows tagged 'insufficient' out of the sorted ranking. - 'Low sample' pill on rows with dataConfidence='low' + tooltip showing the actual sample-health ratio ('45% of expected'). APIs: - /api/citable: returns expectedN + dataConfidence per row. When bench-wide aggregate is 'insufficient', returns value=null + leader=null (refuses to crown a leader from undersized samples). - /api/stat/[slug]: same treatment + drops 'insufficient' rows from the machine-readable ranking array. Per-row sampleHealth + dataConfidence exposed for downstream tooling. Library (citation.ts): - citationCandidates() drops 'insufficient' rows from the leader candidate set. - fieldValue/leader/headlineSentence return null/'insufficient data' when bench-wide is insufficient. - isInsufficient() helper preserved from main (used by hub cards + /api/citable hot path). Effect: bench pages where YAML declares expected_n now visually communicate sample confidence; agents/journalists hitting the citable API can't accidentally quote a leader drawn from 3 data points; the methodology page's 'n >= 1000' claim becomes honest. Co-authored-by: Florent Tapponnier <contact@mobula.io>
Flotapponnier
added a commit
that referenced
this pull request
Jun 26, 2026
…cing (follow-up A) (#741) Completes PR #732 (sample-health system) by surfacing the computed dataConfidence/sampleHealth/expectedN to users + machine-readable consumers. Was dead-code on the data layer until this PR. UI (ledger-table.tsx): - Filter rows tagged 'insufficient' out of the sorted ranking. - 'Low sample' pill on rows with dataConfidence='low' + tooltip showing the actual sample-health ratio ('45% of expected'). APIs: - /api/citable: returns expectedN + dataConfidence per row. When bench-wide aggregate is 'insufficient', returns value=null + leader=null (refuses to crown a leader from undersized samples). - /api/stat/[slug]: same treatment + drops 'insufficient' rows from the machine-readable ranking array. Per-row sampleHealth + dataConfidence exposed for downstream tooling. Library (citation.ts): - citationCandidates() drops 'insufficient' rows from the leader candidate set. - fieldValue/leader/headlineSentence return null/'insufficient data' when bench-wide is insufficient. - isInsufficient() helper preserved from main (used by hub cards + /api/citable hot path). Effect: bench pages where YAML declares expected_n now visually communicate sample confidence; agents/journalists hitting the citable API can't accidentally quote a leader drawn from 3 data points; the methodology page's 'n >= 1000' claim becomes honest. Co-authored-by: Florent Tapponnier <contact@mobula.io>
Flotapponnier
added a commit
that referenced
this pull request
Jun 26, 2026
…cing (follow-up A) (#741) Completes PR #732 (sample-health system) by surfacing the computed dataConfidence/sampleHealth/expectedN to users + machine-readable consumers. Was dead-code on the data layer until this PR. UI (ledger-table.tsx): - Filter rows tagged 'insufficient' out of the sorted ranking. - 'Low sample' pill on rows with dataConfidence='low' + tooltip showing the actual sample-health ratio ('45% of expected'). APIs: - /api/citable: returns expectedN + dataConfidence per row. When bench-wide aggregate is 'insufficient', returns value=null + leader=null (refuses to crown a leader from undersized samples). - /api/stat/[slug]: same treatment + drops 'insufficient' rows from the machine-readable ranking array. Per-row sampleHealth + dataConfidence exposed for downstream tooling. Library (citation.ts): - citationCandidates() drops 'insufficient' rows from the leader candidate set. - fieldValue/leader/headlineSentence return null/'insufficient data' when bench-wide is insufficient. - isInsufficient() helper preserved from main (used by hub cards + /api/citable hot path). Effect: bench pages where YAML declares expected_n now visually communicate sample confidence; agents/journalists hitting the citable API can't accidentally quote a leader drawn from 3 data points; the methodology page's 'n >= 1000' claim becomes honest. Co-authored-by: Florent Tapponnier <contact@mobula.io>
Flotapponnier
added a commit
that referenced
this pull request
Jun 26, 2026
* feat(sample-health): wire UI badge + API surface + insufficient silencing (follow-up A) (#741) Completes PR #732 (sample-health system) by surfacing the computed dataConfidence/sampleHealth/expectedN to users + machine-readable consumers. Was dead-code on the data layer until this PR. UI (ledger-table.tsx): - Filter rows tagged 'insufficient' out of the sorted ranking. - 'Low sample' pill on rows with dataConfidence='low' + tooltip showing the actual sample-health ratio ('45% of expected'). APIs: - /api/citable: returns expectedN + dataConfidence per row. When bench-wide aggregate is 'insufficient', returns value=null + leader=null (refuses to crown a leader from undersized samples). - /api/stat/[slug]: same treatment + drops 'insufficient' rows from the machine-readable ranking array. Per-row sampleHealth + dataConfidence exposed for downstream tooling. Library (citation.ts): - citationCandidates() drops 'insufficient' rows from the leader candidate set. - fieldValue/leader/headlineSentence return null/'insufficient data' when bench-wide is insufficient. - isInsufficient() helper preserved from main (used by hub cards + /api/citable hot path). Effect: bench pages where YAML declares expected_n now visually communicate sample confidence; agents/journalists hitting the citable API can't accidentally quote a leader drawn from 3 data points; the methodology page's 'n >= 1000' claim becomes honest. Co-authored-by: Florent Tapponnier <contact@mobula.io> * feat(seo): /benchmarks/category/[cat] hub pages + BenchmarkGrid lockedCategory (D.2) (#743) Final cherry-pick from dev-archive — adds 7 new SEO-indexable hub pages (one per benchmark category) that target long-tail queries like 'best crypto aggregator benchmarks 2026'. D.2 was deferred from PR D because the BenchmarkCard ↔ MiniChart ↔ BenchmarkCardData type chain had cascading TS errors. Follow-up A (citation.ts isInsufficient → InsufficientCheckInput structural subset) silently fixed those, so this lands cleanly now. Files: - src/app/benchmarks/category/[cat]/page.tsx (new) — server route that renders BenchmarkGrid pre-filtered to one category + breadcrumb + ItemList JSON-LD. generateStaticParams builds the 7 hub routes from CATEGORIES. Each page is fully indexable (canonical, OG, etc.). - src/components/benchmark-grid.tsx (updated) — adds two optional props (lockedCategory, allCategories). Backward compatible: when unset the grid behaves exactly as before (used on /benchmarks). When set (category hub routes), the grid pre-filters and shows the full category pill list so users can hop between hubs. URLs that ship after deploy: - /benchmarks/category/aggregators - /benchmarks/category/bridges - /benchmarks/category/blockchains - /benchmarks/category/trading - /benchmarks/category/wallets - /benchmarks/category/rpcs - /benchmarks/category/nft-apis Each renders the canonical bench grid filtered to that category with its own H1, meta description, and structured data. Sitemap pickup automatic on next regen. Co-authored-by: Florent Tapponnier <contact@mobula.io> * fix(seo): blacklist hex builder slugs and shared composite filter Drop 15 anonymous 0x... HL builder slugs and 5 dead composite slugs from /products listing, sitemap and individual /products/<slug> routes. DEAD_COMPOSITE_SLUGS now lives in providers.ts as the single source of truth; related-providers.ts imports it. Bumps providers cache key v3->v4. --------- Co-authored-by: Florent Tapponnier <contact@mobula.io>
Flotapponnier
added a commit
that referenced
this pull request
Jun 26, 2026
* feat(sample-health): wire UI badge + API surface + insufficient silencing (follow-up A) (#741) Completes PR #732 (sample-health system) by surfacing the computed dataConfidence/sampleHealth/expectedN to users + machine-readable consumers. Was dead-code on the data layer until this PR. UI (ledger-table.tsx): - Filter rows tagged 'insufficient' out of the sorted ranking. - 'Low sample' pill on rows with dataConfidence='low' + tooltip showing the actual sample-health ratio ('45% of expected'). APIs: - /api/citable: returns expectedN + dataConfidence per row. When bench-wide aggregate is 'insufficient', returns value=null + leader=null (refuses to crown a leader from undersized samples). - /api/stat/[slug]: same treatment + drops 'insufficient' rows from the machine-readable ranking array. Per-row sampleHealth + dataConfidence exposed for downstream tooling. Library (citation.ts): - citationCandidates() drops 'insufficient' rows from the leader candidate set. - fieldValue/leader/headlineSentence return null/'insufficient data' when bench-wide is insufficient. - isInsufficient() helper preserved from main (used by hub cards + /api/citable hot path). Effect: bench pages where YAML declares expected_n now visually communicate sample confidence; agents/journalists hitting the citable API can't accidentally quote a leader drawn from 3 data points; the methodology page's 'n >= 1000' claim becomes honest. Co-authored-by: Florent Tapponnier <contact@mobula.io> * feat(seo): /benchmarks/category/[cat] hub pages + BenchmarkGrid lockedCategory (D.2) (#743) Final cherry-pick from dev-archive — adds 7 new SEO-indexable hub pages (one per benchmark category) that target long-tail queries like 'best crypto aggregator benchmarks 2026'. D.2 was deferred from PR D because the BenchmarkCard ↔ MiniChart ↔ BenchmarkCardData type chain had cascading TS errors. Follow-up A (citation.ts isInsufficient → InsufficientCheckInput structural subset) silently fixed those, so this lands cleanly now. Files: - src/app/benchmarks/category/[cat]/page.tsx (new) — server route that renders BenchmarkGrid pre-filtered to one category + breadcrumb + ItemList JSON-LD. generateStaticParams builds the 7 hub routes from CATEGORIES. Each page is fully indexable (canonical, OG, etc.). - src/components/benchmark-grid.tsx (updated) — adds two optional props (lockedCategory, allCategories). Backward compatible: when unset the grid behaves exactly as before (used on /benchmarks). When set (category hub routes), the grid pre-filters and shows the full category pill list so users can hop between hubs. URLs that ship after deploy: - /benchmarks/category/aggregators - /benchmarks/category/bridges - /benchmarks/category/blockchains - /benchmarks/category/trading - /benchmarks/category/wallets - /benchmarks/category/rpcs - /benchmarks/category/nft-apis Each renders the canonical bench grid filtered to that category with its own H1, meta description, and structured data. Sitemap pickup automatic on next regen. Co-authored-by: Florent Tapponnier <contact@mobula.io> * fix(seo): drop dead /alternatives links from perp/PM venue footers + dead compare pair Two SEO regressions feeding 404s into Google's crawl: 1. /products/<slug> for perp/PM venues without an alternatives YAML (lighter, manifold, vertex, grvt, ostium, variational, pacifica, aevo, edgex, paradex, extended, myriad, limitless, aster) hardcoded a /alternatives/<slug> footer link in PmVenueSection and PerpVenueSection. Now gated on loadAlternativeSlugs() membership so only venues with an actual YAML emit the link. 2. /compare hub rendered jupiter-vs-raydium even though raydium has no bench appearances, so /compare/jupiter-vs-raydium 404s downstream. Pre-filter COMPARE_PAIRS against getProvider() so dead pairs disappear from the list and the JSON-LD ItemList until both sides resolve. Plus a defensive guard in buildAlternativesReverseMap() so the emitted alt.slug is always a known live YAML, even if the alt list source ever changes shape. --------- Co-authored-by: Florent Tapponnier <contact@mobula.io>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR A in the dev → main reconciliation series. Pure additive, zero behavior change on benches without
expected_n.What
First cherry-pick from dev: the sample-health classification system. Bench pages and citable APIs can honestly badge undersized samples instead of crowning a leader drawn from 3 data points (low-cadence benches like perp-funding legitimately publish 3 samples per 24h — flat-n thresholds mis-classify them as broken, an E-E-A-T penalty waiting to happen with Google).
health = sampleSize / expected_n:Files
sample-health.ts(classifier + aggregate) + tests +time-constants.tsspec-schema.tsgets optionalexpected_n,types/benchmark.tsgets optionalexpectedN/dataConfidence/sampleHealthmaterialize/load.tscomputes per-provider + aggregate;spec.tspassesexpected_nthrough overlayNOT in this PR (follow-ups)
expected_nper-bench declarations (content drops, one at a time)Risk
Zero behavior change on benches without
expected_n. The field is optional; classifier returns null; no badge renders; no rankings hidden. Existing prod surfaces unaffected.