[tamnd]

ListCrawls cached collinfo.json for six hours and then refused to look at it. Past the TTL the fetch was the only path, so when index.commoncrawl.org went away the command failed with the answer sitting on disk.

Measured during the live outage

index.commoncrawl.org has refused connections for three days. There was a collinfo entry in the cache from 19 hours ago. Before:

ccrawl crawls latest                      # exit 8
ccrawl paths warc -n 1                    # exit 8
ccrawl columnar urls --tld gov -n 1       # exit 8
ccrawl download warc -n 1 --dry-run       # exit 8

After, all four exit 0, each with one line on stderr:

crawls: the index server is unreachable, using the crawl list cached 19h0m0s ago; pass -c to name a crawl instead of resolving it
CC-MAIN-2026-30

$ ccrawl columnar urls --tld gov -n 1 -o url
https://www.13thjudicialil.gov/

Every one of those reads its real data from data.commoncrawl.org, which answered 200 throughout. They only touched the index server to turn the word latest into a crawl ID. The columnar one is the quick start printed in ccrawl --help.

With the cache empty the run still fails, which is the point:

$ CCRAWL_CACHE_DIR=/tmp/emptycache ccrawl crawls latest; echo $?
8

An exit code this uncovered

search keeps going past index pages it cannot read, and a run that emitted nothing came back as exit 1 whatever the reason. That path was mostly unreachable during a full outage because the crawl list failed first. Now that the crawl list resolves from cache, a total outage lands there, and search 'example.com/*' exited 1 instead of 8, which is the bug #131 fixed coming back through a different door.

A run that lost everything to the transport now exits 8. A run where any part of the loss was a status the server sent stays 1: a supervisor that backs off and retries against a server which is up and refusing never stops. Measured live, search 'example.com/*' is back to exit 8.

internal/fakecc grew HangupPage so a test can tell the two apart end to end. DeadPage answers 503, HangupPage drops the connection with no status line.

Tests

Four in ccrawl: the stale fallback resolves and warns with the age in it, no cached copy still fails, a reachable server still beats a stale cache so a new crawl is not hidden behind a cache clear, and the age wording.

Three in cli, end to end against the fake Common Crawl: a hangup on the only page exits 8, a 503 on the only page exits 1, and a run that lost one of each exits 1.

Checks

gofmt, go vet, go test ./..., golangci-lint run ./... (0 issues), scripts/docs-drift.sh (406 flag mentions, reference matches the binary), scripts/coverage.sh (ccrawl 66.5, cli 52.7, every package above its floor).

Found on the way and filed separately, not fixed here: #138, --data-dir does not move the cache dir although the reference says it does.

Closes #137