[tamnd]

The last unticked line on the v0.3.0 acceptance list in #8:

a record fetched from a disallowed path carries robots.allowed: false and the rule that matched it

It was not implemented. A grep for robots_allowed, RobotsAllowed and robots.allowed across goodread/ and cli/ came back empty.

What this does

Envelope gains an optional robots block, and a record built from a page robots.txt asked us not to read carries it:

"robots": {
  "allowed": false,
  "path": "/book/reviews/2767052",
  "rule": "Disallow: /book/reviews/",
  "source": "https://www.goodreads.com/robots.txt"
}

allowed is not omitempty, so a consumer reading robots.allowed gets a false rather than a missing key, and rule quotes the line from the file rather than paraphrasing it.

A record off an allowed surface carries nothing, so the absence means as much as the presence, and neither depends on the reader knowing which flags the run used.

The client remembers the verdict on every path the override let through, and each record fetcher stamps what it built. Stamping is a no-op for an allowed page, which is what lets every fetcher call it unconditionally: a surface that becomes disallowed on a later reading of robots.txt starts carrying the note with no further code.

Reviews carry the block on the row rather than on an envelope. A review is a row and has no envelope, and reviews --all returns the book page's sample and the walked pages in one list, so a note on the row is the only thing that keeps them distinguishable after the merge.

The bug this turned up

The v0.3.0 record fetchers built URLs with the package helpers, which hardcode BaseURL. So a test that pointed the client at a local server with SetBaseURL still fetched goodreads.com for real. The first run of the new test did exactly that and the second one came back with a WAF challenge, which is how it was noticed. They now go through c.site(), which is a no-op in production because c.base is BaseURL.

Tests

Three, all against the golden captures rather than fixtures written to pass.

• TestDisallowedReadCarriesTheNote walks /book/reviews off the s12 capture and checks every row carries the note with the real rule.
• TestAllowedReadCarriesNoNote reads the s1 book page with --no-robots on, and checks nothing on the record claims otherwise.
• TestEveryRecordFetcherStamps parses the package and requires every Get*Record method to either stamp or be built out of others that did, so a fetcher added later cannot quietly skip it.

Docs

The envelope table in the output reference, the --no-robots section of the robots guide, and the release notes in both docs/ and .goreleaser.yaml.

Gate: gofmt -l . && go vet ./... && go test ./... clean.