feat(hf): daily snapshot publisher to Hugging Face dataset - #632
Merged
Conversation
Publishes a daily snapshot of /api/citable + /api/stat to huggingface.co/datasets/OpenChainBench/benchmarks as three Hive-partitioned Parquet tables: headlines, providers, timeseries. Quorum guard refuses to publish a degraded snapshot. Schema is versioned and additive-only. Slack alerts on abort/crash. Validated against prod /api/citable: 26 headlines, 254 provider rows, 1838 timeseries points, ~50KB total per snapshot.
Flotapponnier
added a commit
that referenced
this pull request
Jun 22, 2026
Same files as #632 on dev, brought to main so the GitHub Actions cron schedule can fire (workflow_dispatch and on.schedule both require the workflow file to live on the default branch). No runtime code change. Only adds: - scripts/hf_publisher/* (Python daily snapshot script + tests) - .github/workflows/hf-publish.yml (cron + manual dispatch) - .gitignore (.venv exclusion)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Publishes a daily Parquet snapshot of every public OCB benchmark to
huggingface.co/datasets/OpenChainBench/benchmarks.
Goal: backlink Tier 1 (HF dataset pages indexed by Google + scraped by
ChatGPT/Claude/Perplexity), plus a queryable archive for journalists
and academic users.
How it works
Daily cron at 00:00 UTC runs
scripts/hf_publisher/publish.py:/api/citableand/api/stat/<slug>from prod<50%benches are live or count<8headlines/(26 rows/day, ~15KB): 1 row per (slug, day) with leader + valueproviders/(254 rows/day, ~20KB): 1 row per (slug, provider, day) with p50/p90/p99timeseries/(1838 rows/day, ~15KB): 1 row per 24h sparkline pointTotal ~50KB per snapshot, ~18MB/year. Compressed zstd.
Validation
Dry-run executed against live prod:
```
quorum ok: 26/26 live
wrote headlines part-0.parquet rows=26 size=15KB
wrote providers part-0.parquet rows=254 size=20KB (26 leaders)
wrote timeseries part-0.parquet rows=1838 size=15KB
```
Spot-checked content: aggregator-head-lag leader=Mobula 681s, bridge-quote-latency leader=Mobula 342ms, bridge-fee leader=Near Intents, all coherent with /api/citable.
Tests: 15/15 pass offline (quorum guard, row builders, partitioning, template substitution).
Durability features
Required GitHub secrets
Test plan
After merge to dev: