Repository navigation
M1: obey robots.txt on every request, add --no-robots - #18
Merged
Merged
Conversation
v0.2.0 fetched /search, /review/list and /review/list_rss, all three disallowed by goodreads robots.txt. It never checked. This adds the check and puts it somewhere it cannot be walked around. The check lives in Client.Fetch, not in Client.Do. Do would only cover call sites ported to the new op registry, and the defect was that call sites built their own URLs. A check you can skip by not calling the right function is not a check. robots.go parses the User-agent: * group with longest-match-wins and Allow beating Disallow at equal length, which is what the two /work exceptions depend on: Allow: /work/editions is longer than Disallow: /work, so editions and quotes stay readable and /work does not. Matching runs on path plus query, since Disallow: /*reviewFilters can only ever match a query string. Rules are fetched on first use and cached for an hour. A fetch failure is exit 8 and nothing is attempted, because a crawler that carries on when it cannot read the rules is the thing this change exists to stop. surfaces.go registers every path the tool knows how to read, so the policy test can walk the list rather than trusting that somebody remembered. TestNoDisallowedByDefault asserts the declared and the live rules agree in both directions, so a rules change on the site fails loudly instead of silently widening what we fetch. --no-robots restores everything v0.2.0 could do. It is settable by the flag only, never by config and never by an environment variable, and three tests hold that. It warns once on stderr, and the pace floor and any published Crawl-delay still apply, because ignoring a stated rate is the part that actually causes harm. Measuring the live file found something the spec had guessed wrong: /book/auto_complete is not disallowed, is public JSON, needs no key, and returns bookId, workId, title, numPages, avgRating and ratingsCount per hit. So search keeps working with no flag, and it returns more than it used to. Cookies are gone. No login, no Cookie header, no browser driver, no challenge solving, and TestNoCookieHeader, TestNoBrowserDependency and TestNoChallengeHandling keep it that way. The AWS WAF challenge header is reported as blocked, not solved. Part of #8. Closes #9.
…written TestNoRobotsNotImplied walks every flag on the command tree, sets each one, and asserts none of them turns the override on. The failure it guards against is not somebody wiring --no-robots to --force on purpose. It is a flag added later that sets a whole config struct at once and quietly makes the override reachable without anybody typing it. TestNoConcurrency pins the transport to one connection per host and checks that Workers=64 does not widen it. The pace floor is about the interval between requests, this is about how many are in flight, and a 1s floor with eight connections is eight requests a second as far as the site is concerned. Two smaller ones while I was there: TestNoRobotsIsNotHidden, because an override nobody can see in --help is one people find in a blog post without the paragraph explaining what it means, and TestCookiesFlagIsGone.
This was referenced Aug 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #9. Part of #8.
What was wrong
v0.2.0 fetched
/search,/review/listand/review/list_rss.All three are disallowed by goodreads
robots.txtand the tool never looked.What this does
Adds the check and puts it in
Client.Fetchrather thanClient.Do.Dowould only cover call sites ported to the new op registry, and the defect was precisely that call sites built their own URLs.A check you can skip by not calling the right function is not a check.
goodread/robots.goparses theUser-agent: *group with longest match wins andAllowbeatingDisallowat equal length.That precedence is load bearing:
Allow: /work/editionsis longer thanDisallow: /work, so editions and quotes stay readable and the bare work page does not.Matching runs on path plus query, because
Disallow: /*reviewFilterscan only ever match a query string.Rules are fetched on first use, cached an hour, and a fetch failure is exit 8 with nothing attempted.
There is no compiled in fallback copy, because a stale copy that says yes is worse than no answer.
goodread/surfaces.goregisters every path the tool can read.TestNoDisallowedByDefaultwalks the registry and asserts the declared and the live rules agree in both directions, so a rules change on the site fails loudly instead of quietly widening what we fetch.--no-robotsEverything v0.2.0 could do still works with the flag.
It is settable by the flag only, never by config and never by an environment variable, and
TestNoRobotsNotInConfig,TestNoRobotsNotInEnvandTestNoRobotsNotImpliedhold that.It warns once on stderr, and the pace floor and any published
Crawl-delaystill apply either way, since ignoring a stated rate is the part that actually causes harm.One thing the spec had wrong
Capturing the live file settled a question
~/notes/Spec/3008had guessed at./book/auto_complete?format=json&q=is not disallowed by any rule, is public JSON, carries no key, and returnsbookId,workId,title,numPages,avgRating,ratingsCountand the author per hit.So
goodread searchkeeps working with no flag, and it returns more than it used to.workIdon every hit is the field that makeseditionsandquotesreachable without a second read.Registered as s14, guarded by
TestSuggestIsAllowed, and the spec is corrected.Removed
goodread/cookies.goand the--cookiesflag.No login, no
Cookieheader, no browser driver, no challenge solving.TestNoCookieHeader,TestNoBrowserDependencyandTestNoChallengeHandlingkeep it that way, and the AWS WAF challenge header is reported as blocked rather than solved.New command
Verified live against goodreads.com
goodread robots -f tableprints 29 rules with the/work/editionsexception attributed correctlyrobots check /work/editions/2792775exits 0robots check '/book/show/2767052?reviewFilters=%7B%7D'exits 7robots check '/search?q=x'exits 7, and exits 0 with--no-robotsgoodread search "hunger games"returns three real results with no flaggoodread shelf 1is refused with the rule named, and works after one stderr warning with--no-robotsGate
gofmt -l .clean,go vet ./...clean,go test ./...ok forcliandgoodread.32 new tests, all against the captured
robots.txtandhttptestservers, none against the live site.