Skip to content

feat(hf): daily snapshot publisher to Hugging Face dataset - #632

Merged
Flotapponnier merged 1 commit into
devfrom
feat/hf-dataset-publisher
Jun 22, 2026
Merged

feat(hf): daily snapshot publisher to Hugging Face dataset#632
Flotapponnier merged 1 commit into
devfrom
feat/hf-dataset-publisher

Conversation

@Flotapponnier

Copy link
Copy Markdown
Collaborator

What

Publishes a daily Parquet snapshot of every public OCB benchmark to
huggingface.co/datasets/OpenChainBench/benchmarks.

Goal: backlink Tier 1 (HF dataset pages indexed by Google + scraped by
ChatGPT/Claude/Perplexity), plus a queryable archive for journalists
and academic users.

How it works

Daily cron at 00:00 UTC runs scripts/hf_publisher/publish.py:

  1. Fetch /api/citable and /api/stat/<slug> from prod
  2. Quorum guard: refuse to publish if <50% benches are live or count <8
  3. Build 3 Hive-partitioned Parquet tables:
    • headlines/ (26 rows/day, ~15KB): 1 row per (slug, day) with leader + value
    • providers/ (254 rows/day, ~20KB): 1 row per (slug, provider, day) with p50/p90/p99
    • timeseries/ (1838 rows/day, ~15KB): 1 row per 24h sparkline point
  4. Stage README, CITATION.cff, LICENSE, JSON schemas, examples (pandas/polars/duckdb)
  5. Push to HF via huggingface_hub
  6. Slack alert on abort/crash

Total ~50KB per snapshot, ~18MB/year. Compressed zstd.

Validation

Dry-run executed against live prod:

```
quorum ok: 26/26 live
wrote headlines part-0.parquet rows=26 size=15KB
wrote providers part-0.parquet rows=254 size=20KB (26 leaders)
wrote timeseries part-0.parquet rows=1838 size=15KB
```

Spot-checked content: aggregator-head-lag leader=Mobula 681s, bridge-quote-latency leader=Mobula 342ms, bridge-fee leader=Near Intents, all coherent with /api/citable.

Tests: 15/15 pass offline (quorum guard, row builders, partitioning, template substitution).

Durability features

  • Schema versioning: `schema_version` int column in every row, additive-only contract documented in README
  • Quorum guard: bad source feed = no publish, previous snapshot stays as truth
  • Idempotent: same snapshot_date overwrites itself
  • Tests gate every PR: schema regression caught in CI before HF push
  • Slack alerts: `SLACK_WEBHOOK_URL` secret pinged on abort/crash
  • HF git history: each snapshot = 1 commit, rollback via `git revert` on HF

Required GitHub secrets

  • `HF_TOKEN` (write scope on dataset repo)
  • `SLACK_WEBHOOK_URL` (optional, for ops alerts)

Test plan

After merge to dev:

  • Manually trigger via Actions tab with `dry_run: true` to validate the env wires up
  • Re-trigger with `dry_run: false` to ship the first real snapshot
  • Verify dataset page at huggingface.co/datasets/OpenChainBench/benchmarks shows preview
  • Let cron run once at 00:00 UTC tomorrow, verify a second partition lands
  • After 7 days, merge dev -> main (the workflow file lives in CI, not in the runtime path, so the merge is trivial)

Publishes a daily snapshot of /api/citable + /api/stat to
huggingface.co/datasets/OpenChainBench/benchmarks as three
Hive-partitioned Parquet tables: headlines, providers, timeseries.

Quorum guard refuses to publish a degraded snapshot. Schema is
versioned and additive-only. Slack alerts on abort/crash.

Validated against prod /api/citable: 26 headlines, 254 provider rows,
1838 timeseries points, ~50KB total per snapshot.
@Flotapponnier
Flotapponnier merged commit c00721a into dev Jun 22, 2026
1 check passed
Flotapponnier added a commit that referenced this pull request Jun 22, 2026
Same files as #632 on dev, brought to main so the GitHub Actions
cron schedule can fire (workflow_dispatch and on.schedule both
require the workflow file to live on the default branch).

No runtime code change. Only adds:
- scripts/hf_publisher/* (Python daily snapshot script + tests)
- .github/workflows/hf-publish.yml (cron + manual dispatch)
- .gitignore (.venv exclusion)
@Flotapponnier
Flotapponnier deleted the feat/hf-dataset-publisher branch July 17, 2026 14:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant