Repository navigation
M7: the crawler #15
Description
Spec: ~/notes/Spec/3008/05_commands.md section 6.
Single connection, 2s pace, no parallelism flag.
A crawler that can be told to open ten connections will be, and the site does not get a say.
The crawl refuses to start with --no-robots unless --yes is also given, because a crawl is the one place where an override multiplies.
--dry-run prints the plan and the request count and reads nothing, which is how you find out a crawl is twelve hours before starting it.
Checklist
- seeds as URIs, repeatable, plus
--seed-fileand--from-sitemap - refuses to start with
--no-robotsunless--yesis also given - single connection, 2s pace, no parallelism flag
- resumable, state in the store
-
--dry-runprints the plan and the request count and reads nothing - progress output with counts per type, requests, cached, errors, elapsed
Part of #8.
Activity
tamnd commentedon Aug 15, 2026
Done in #26, merged as 7687357.
Three decisions worth writing down.
The parallelism flag is not offered, rather than offered with a default of one.
The v0.2.0 command had a worker pool and a
--workersflag, and that is the part that got deleted rather than defaulted.A crawler that can be told to open ten connections will be.
The pace is the client's shared limiter, so a crawl is now structurally incapable of being faster than
goodread book.The frontier is keyed by URI and not by URL.
The sitemap spells one book as both
/book/show/2767052and/book/show/2767052-the-hunger-games, and the old URL keyed queue read it twice and called that two books.Reaching a URI again at a shallower depth lowers the recorded depth, because the budget is measured from the seed and the shortest path is the honest distance.
Expansion goes through the edge table, not through the HTML.
The crawler reads a page,
Store.Putfolds it into nodes and edges for free, andexpandenqueues the neighbours of the node that was just written.So the crawler does not need to know what links a book page carries, and a surface that starts naming a new relationship widens the crawl without this file being touched.
A place, a character and an award have no page of their own, so
URIToURLreturns not-ok for them and they are skipped rather than turned into a URL that 404s.On
--no-robots: the refusal is before anything is read and before the frontier is touched, and the message says the override applies to every request the crawl makes rather than to one page.--yesis the second flag because that is where the multiplication gets acknowledged.Tests are against the golden captures over a local server, with a rewriting transport rather than
SetBaseURL, so the URL builders being exercised are the real ones.The resumability test runs a crawler with
--max 1, then builds a second crawler over the same store and checks it finishes the job rather than starting it.