Repository navigation
ccrawl search --limit N fails with "Cdx page 0: stop." once the limit is hit #105
Description
ccrawl search <pattern> --limit N prints an error and exits 1 as soon as it has emitted N records:
$ ccrawl search 'commoncrawl.org/*' -c 5 --limit 5 -o url
http://commoncrawl.org/
https://commoncrawl.org/
https://commoncrawl.org/
ERROR
Cdx page 0: stop.
The records are correct. The error is not real, it is the framework's row-limit sentinel escaping as a failure.
Cause
CDXStream wraps anything the callback returns with the page it came from:
if err != nil && err != errStop {
return fmt.Errorf("CDX page %d: %w", page, err)
}kit stops an emit loop by returning an unexported sentinel and comparing it by identity on the way back (kit/op.go:288, err != errStop). %w preserves the chain but not the identity, so once ccrawl adds "CDX page 0: " to it kit no longer recognises its own stop signal and renders it as an error.
cli/export.go already hit this and worked around it with errors.Is. cli/ops.go, which is where search lives, cannot, because kit's sentinel is not exported.
Fix
CDXStream should hand the callback's error back exactly as it was raised, and keep the page context for errors that are actually about the page.
Found while
Measuring the shared rate limit in #61.
Activity
tamnd commentedon Aug 12, 2026
Fixed in #106, with the lint follow-up in #107.
CDXStreamnow carries the callback's error out through a closure variable so it reaches the caller exactly as it was raised. The page context stays on errors that are actually about the page, which is what it was for. The stale workaround comment incli/export.gois gone.The CDX paging loop had no test coverage at all, so
cdxBasemakes the server root swappable andccrawl/cdx_test.godrives it against an httptest server.TestCDXStreamReturnsTheCallbackErrorUnwrappedcompares by identity rather thanerrors.Is, becauseerrors.Ispasses straight through the wrap and would hide this exact bug. Against the old code it fails with the string a user would see:The others pin the limit stopping the stream without fetching the next page, a failing page still naming itself, the crawl ID getting stamped on each record, a zero-page response still having page 0 read, and 404 reading as no captures rather than as an error, which is what lets
searchexit 3 for a query the index has nothing for.Live check still outstanding. index.commoncrawl.org has been refusing connections at the dial for most of today (
dial tcp 54.237.141.66:443: connect: connection refused, single A record, refused for curl too, with one brief window where it answered). Soccrawl search --limit 5has not been re-run against the real index yet. The unit test reproduces the exact symptom, so the fix is verified at the level that matters, but the one-line live run is worth doing when the index is back.