[tamnd]

Closes #51.

Frontier.seen was a map[string]struct{} and the queue was a slice. That works right up to the size the frontier is actually for: the gao seed set is around 280M URLs, and a Go map with 280M string keys is tens of gigabytes of live heap and a garbage collector that never finishes.

The state is a SQLite file now, modernc.org/sqlite so there is still no cgo. One table for the frontier keyed on a 16 byte URL hash, one for per host politeness, and a staging table in front of the frontier.

The part that took the work

The obvious version of this is slow in a way that does not show up until it matters. Inserting discovered URLs straight into the frontier writes to a random page of a B-tree that is far larger than any page cache, and the admit rate falls off a cliff as the table grows.

rows so far	admit rate, inserting directly
200k	170k/s
3M	2.9k/s
4M	still falling

Sorting each batch by key first helps a lot and is still not enough, because every batch sweeps the whole tree regardless of how few rows it carries.

So admissions go to a staging table, which is a rowid table written in increasing rowid order, and an append to the right hand edge of a B-tree costs the same at four million rows as at four hundred. They are merged into the frontier with one INSERT OR IGNORE ... ORDER BY url_hash, which walks the frontier left to right once and reads and dirties each leaf page once. The merge threshold scales with what is already queued, so the number of sweeps over a crawl stays small instead of growing with it.

The same locality argument applies to the other two write paths, so completions and politeness clocks are buffered and written in key order too.

Pops claim 512 rows in one query and serve them from memory, so the pop rate is a memory operation with a query amortised over hundreds of it. A claimed URL whose host is inside its politeness delay is written back with next_at set to when the host is next eligible, rather than rescanned.

Why there is no Bloom filter

The issue asks for a Bloom filter in front of the seen check, and I did write one first. It is the wrong structure for this position and the test run says so: a Bloom filter's error is a false positive, which in front of a seen check means reporting a brand new URL as already known, and the frontier then drops it. Sized at the conventional one percent it silently lost 52 of 50000 URLs.

One page in a hundred missing from a crawl, with nothing downstream able to tell, is not a trade a data structure should make on the operator's behalf. So the cache is exact for what it holds and forgets the coldest keys instead of lying about them. A forgotten key costs one wasted row in the next merge, where the primary key drops it, and nothing else. Memory is bounded the same way a filter would have bounded it, by a configured size.

Restart

A claim is a state on the row. Rows a dead run left claimed go back to pending on open, because a claim that outlived its process is a URL that was fetched or was not and only one of those readings is safe. Rows marked done stay done, which is what stops a restart refetching. Staged rows that never merged are merged by the next flush.

Completions are buffered up to SyncEvery, so a clean stop refetches nothing and a kill refetches at most one claim batch. That buffering is worth about four times the pop rate. SyncEvery: 1 gives back the exactness for callers that want it, and Sync() is there for a caller that has just been told to stop.

Numbers

10M URLs across 2M hosts, one laptop disk, default config except a 4M entry seen cache:

admit    2000000   125021/s  heap  172 MB  sys  203 MB
admit    4000000    83941/s  heap  203 MB  sys  342 MB
admit    6000000    74520/s  heap  382 MB  sys  463 MB
admit    8000000    63759/s  heap  377 MB  sys  537 MB
admit   10000000    38407/s  heap  368 MB  sys  675 MB   database 1.18 GB
pop      2000000     8253/s
pop      4000000     8827/s
pop      6000000     9179/s

Pop does not decay, which is the property that matters, and every one of the 10M URLs came back out. At 200k the same code does 170k/s admit and 42k/s pop in 14 MB of heap.

Reproduce with CCRAWL_SCALE_URLS=10000000 go test ./ccrawl -run TestFrontierScale -v -timeout 2h.

Done when

• 280M URLs admitted with RSS under 4 GB. Not run. It needs around 33 GB of database and this machine has 23 GB free, so the honest answer is that the number is an extrapolation and not a measurement. What is measured is that RSS is bounded by the seen cache and the page cache rather than by the number of URLs, both of which are configured sizes, so it is flat in the frontier's size by construction rather than by luck. At 10M it is 675 MB.
• Pop rate at or above 5k URLs per second. 9.2k/s at 10M rows and not decaying, 42k/s at 200k.
• A restart resumes with no duplicate fetches. Covered by TestFrontierResumesWithoutRefetching, TestFrontierReclaimsClaimsLeftInFlight and TestFrontierDedupsAcrossRestart. Exact across a clean stop, bounded by SyncEvery across a kill.

Scope

Library only. Nothing calls NewFrontier outside the tests yet, which is why the redesign was free. ccrawl crawl run is #54 and comes last on purpose.

Gates: gofmt, go vet, go test -race ./... -count=1, golangci-lint run clean, scripts/docs-drift.sh clean. No CLI surface changed, so there is nothing for the reference to drift from.