[tamnd]

Closes #52.

The old parser matched literal path prefixes, knew two user agent names, and read a 5xx on /robots.txt as permission to crawl. The last one is the one the issue is filed as a bug for. A site whose robots endpoint is failing cannot tell us to stop, and crawling it anyway is how a crawler ends up in a block list. RFC 9309 section 2.3.1.4 calls an unreachable robots.txt a complete disallow, and so does this now, remembered for five minutes so the host is crawled again as soon as it recovers.

What is in

• * and $ wildcards, longest match wins, an allow breaks a tie of equal length
• case insensitive product token matching, most specific group wins, the wildcard group is a fallback and never merges with a specific one, groups naming the same crawler are followed as one
• an empty group naming a crawler still counts as that crawler's group, so it does not fall back to *, which is the quxbot case in the RFC's example
• percent encoding normalised on both sides, since the comparison is octet by octet
• Sitemap lines collected and exposed, and they do not interrupt the group they sit in
• the 500 KB parse limit, with the rest discarded rather than treated as an error
• Crawl-delay honoured, which is not in RFC 9309 but is still what sites write

Status handling had to get more careful than a bool

A 403 means there is no robots.txt to read and the site is open. A dead host means assume everything is off limits. The HTTP client was collapsing both into the same error, because it retries 403 and 5xx alike and reports the exhausted retry as a plain error. getStatus hands back the last response instead, so robots can tell a refusal from a silence.

Crawl-delay needed somewhere to land

Parsing a delay is worthless if nothing enforces it. Frontier.Hold pushes a host's next eligible time out and never brings it forward. Pop reserves the crawl's own delay when it hands a URL out, because at that moment nobody has read the host's robots.txt yet, and the fetcher extends it once it knows. TestCrawlDelayHoldsTheHostInTheFrontier pops a URL, applies a parsed Crawl-delay: 10, and checks the same host stays shut at 1, 5 and 9 seconds and opens at 10.

Done when

• the RFC 9309 example suite passes. The section 2.2.2 match table and the section 5.2 example file are both transcribed, including the four per-bot verdicts the RFC spells out for that file.
• a host returning 503 on /robots.txt does not get crawled. TestFetchRobotsDisallowsWhenUnreachable covers 500, 502 and 503, and TestFetchRobotsDisallowsWhenHostIsDown covers nothing listening at all.
• Crawl-delay: 10 produces at least 10 seconds between requests to that host.

Real data

Hand written cases prove the parser matches the RFC as I read it, which is not the same as being right. scripts/robots-corpus.py fetches robots.txt from 31 live sites, derives probe paths from each file's own rules, and records what protego, the parser Scrapy uses, concludes about every one. TestRobotsAgainstRealSites replays that corpus: 5949 probes, zero disagreements, and the crawl delay and sitemap count match on all 31 files.

431 probes are not compared, and they are worth spelling out rather than hiding. Every one of them is protego getting it wrong the same way: it percent encodes the URL and not the pattern, so Disallow: /gp/aw/help/id=sss does not match /gp/aw/help/id=sss, and the same for a %3D spelling of it. Two smaller variants of the same habit, a trailing ? that urlparse drops and a leading // it reads as a host, account for the rest. The filter lives in the corpus script next to the tool it is about, with the reasoning written down.

Live check through the CLI, not just tests:

ccrawl crawl fetch https://en.wikipedia.org/wiki/Main_Page --robots     # 200
ccrawl crawl fetch https://en.wikipedia.org/wiki/Special:Search --robots # refused

Not in this PR

Nothing enforces robots on a crawl yet, because nothing drives a crawl yet. Wiring the cache and the delay into a loop is #54.