The Hacker News archive, supercharged. Live at hackernews.raindb.io.
An independent, read-only interface to the HN archive running entirely on RainDB: every item is an immutable droplet, listings/search/trends are live SQL over the whole archive (DuckDB on Parquet via Periscope), and "ask the archive" is an AI agent that answers questions grounded in real threads with mandatory citations.
Not affiliated with Y Combinator. Content is user-generated on news.ycombinator.com, synced from the official HN API; every item links back to its source.
| Surface | RainDB feature |
|---|---|
| front page | live API ids hydrated from archive pointer reads (O(1) per item) |
| new / all-time top | one SQL statement over the entire archive |
| search | full-archive LIKE scan -- no search infrastructure at all |
| trends | mentions-per-year aggregation across every item ever |
| sql | public read-only SQL explorer with wall-clock timings |
| ask the archive | agent loop + SQL tool + thread tool, SSE-streamed, [item:ID] citations |
| item pages | whole thread in ONE prefix walk (denormalized rootId index) |
| sync | sequential item ids = trivial resumable cursor; re-writes are new revisions |
official HN API ──backfill.mjs──> data/chunks/*.json ──ingest.sh──> hn-items formation
(droplets + indexes)
Browser ── /api/* ──> Lightning bolt (server/) |
front page ───────── readLatest x30 (by-id pointer index) <──────────┤
thread view ──────── listKeys walk (by-root index) <─────────────────┤
lists/search/trends─ sql.query (Periscope: DuckDB over Parquet) <────┤
ask the archive ──── runAgent + search_archive + get_thread tools <──┘
One formation: hn-items (story/comment/job/poll/pollopt discriminated
by itemType). See formations/hn-items-config.json -- the by-update
index is what feeds Periscope its pooling cursor (do not remove it).
Prereqs: raindb-cli (raindb.io), Node >= 20.
git clone https://github.com/gignit/raindb-hackernews && cd raindb-hackernews
raindb-cli user register # or: user login
raindb-cli group create <org>
raindb-cli tenant create hn --group <org>
scripts/setup.sh --profile core.<env>.hn
# load data (resumable; run big ranges under nohup/runner)
node scripts/sync/backfill.mjs --latest 50000 # newest 50k items
bash scripts/sync/ingest.sh # chunks -> droplets
bash scripts/sync/verify.sh # SQL counts
# keep it current (cron this)
node scripts/sync/sync-latest.mjsThe full archive is ~48M items. backfill.mjs --from 1 --to 48000000
is the same code -- ranges are resumable, the manifest tracks progress,
and the scripts are Node-stdlib-only so you can run them next to the
API (e.g. on an AWS instance in-region with RainDB) for max throughput.
Same model as raindb-starter:
cd client && npm run dev gives Vite HMR against the LIVE bolt;
git commit auto-deploys the server (post-commit hook); npm run deploy:client ships the UI when you're ready.
Code: MIT. Archive content: user-generated Hacker News submissions,
fetched from the official API; each item carries its canonical
news.ycombinator.com source link and author attribution. If you are a
rights holder and want something changed, open an issue.