v0.3.0: obey robots.txt with an explicit override, and read the Apollo cache instead of the rendered page

v0.3.0 spec is written and lives in ~/notes/Spec/3008, seven documents.
This issue is the checklist for the whole version.
Each milestone below gets its own issue and its own PR, and this list gets ticked as they land.

Why this version exists

v0.2.0 builds three URLs that robots.txt disallows: /search, /review/list/ and /review/list_rss/.
It does not fetch robots.txt at all, so it is not ignoring the rules, it does not know they exist.
That is the defect. The tool has no opinion when it should have a default and a way to override it.

v0.3.0 fetches, parses and obeys robots.txt by default, and --no-robots turns that off for somebody who has decided it is their call.
Nothing is removed. Three commands need the flag.

The second half of the version is extraction.
Every Goodreads page ships its whole Apollo GraphQL cache inline in __NEXT_DATA__, 129 typed entities on a book page, including fields that are never rendered: ratingsCountDist, textReviewsLanguageCounts, places, characters.
parse.go is 1,033 lines of CSS selectors reading the rendered output of that same data.

M1 robots

M2 extraction

M3 model

M4 commands

M5 store

SQLite, tables per node type plus edges
full record in json, columns as an index over it
isbn13 and asin indexed
FTS5 over title, description and author name
merge on upsert, field by field, newest non absent wins
a test that a quick read after a full read does not delete the editions list
goodread find, goodread lookup, goodread query

M6 graph

both id spaces kept, legacy and KCA, neither derived from the other
gr: URI space, built from the legacy id
slug() specified once and tested, with a note that changing it is a breaking change
the twelve edges, each carrying its surface and timestamp
contributed_by carries the role, in_series carries the position
traversal with --depth, defaulting to 1, asking for --yes above 500 nodes
JSONL export
RDF export, schema.org where it fits and grv: where it does not
schema:exampleOfWork for the book to work edge

M7 crawl

seeds as URIs, repeatable, plus --seed-file and --from-sitemap
refuses to start with --no-robots unless --yes is also given
single connection, 2s pace, no parallelism flag
resumable, state in the store
--dry-run prints the plan and the request count and reads nothing
progress output with counts per type, requests, cached, errors, elapsed

M8 mcp

goodread mcp over stdio
the eleven read tools, argument names matching the flags
no search, no shelf, no reviews, and --no-robots has no effect on the server
TestMCPHasNoDisallowedTools
startup and --help both say why

M9 release

README rewritten, one sentence per line
everything under docs/ rewritten the same way
goreleaser dockers_v2 publishing ghcr.io/tamnd/goodread, tagged 0.3.0 and :latest
the container image actually pulled and run before the issue closes
govulncheck against the tag, not only against the branch
handwritten release notes with the generated changelog below them
v0.3.0 tagged

Blocked

homebrew and scoop publishing, waiting on HOMEBREW_TAP_GITHUB_TOKEN and SCOOP_BUCKET_GITHUB_TOKEN as repo secrets

Acceptance

with default settings, no request the tool can construct is disallowed by the live robots.txt, proven by a test walking every registered op against a captured copy and a fetched one
with --no-robots, every v0.2.0 command works
--no-robots cannot be set by config, by env, or by another flag
the rate floor holds with --no-robots set
a record fetched from a disallowed path carries robots.allowed: false and the rule that matched it
cookies.go is gone and no code path sends a Cookie header
level 3 field count is zero, or the release notes say what is left and why

Milestone issues are open and linked to the v0.3.0 milestone.

M	Issue	Ships
M1	#9	robots.txt fetched, parsed, enforced, and the `--no-robots` override
M2	#10	`__NEXT_DATA__` and the Apollo cache
M3	#11	the record types, provenance, honest absence
M4	#12	every command on the new extractor
M5	#13	the SQLite store
M6	#14	identity, edges, export
M7	#15	the crawler
M8	#16	the MCP server
M9	#17	docs, packaging, the tag

Order is deliberate. M1 first means the defect that motivates this version is fixed in the first shippable increment, and everything after it is built on a client that cannot construct a disallowed request without being told to.

Starting on M1 now.

M1 is done and up as #18. Ticked above.

Two notes worth carrying into the rest of the milestones.

Where the check lives is the load bearing decision.
It is in Client.Fetch, not Client.Do. Do would only cover call sites that had been ported to the op registry, and the defect this version exists for was that three call sites built their own URLs and nothing sat between them and the network. So M2 through M8 can port call sites to Do at whatever pace suits them without leaving a gap while they do it.

The spec was wrong about search, and the correction is good news.
/book/auto_complete?format=json&q= is not disallowed, is public JSON, needs no key, and returns bookId, workId, title, numPages, avgRating, ratingsCount and the author per hit. So goodread search needs no flag, and it returns more than the scraped version did. workId on every hit is the one that matters downstream: it makes editions and quotes reachable without a second read, which changes what M7 crawl can do cheaply. Registered as s14. ~/notes/Spec/3008/01_surfaces.md, 00_overview.md and 05_commands.md are corrected.

Starting M2, #10, extraction.

M1: obey robots.txt on every request, add --no-robots (#18)

feat(extraction): read the Apollo cache the page already ships (#19)

M3 is in, merged as #20.

The record model, the mapper from the extractor onto it, goodread book <id> --check and goodread verify --sample N.

The v0.2.0 types moved out of the way rather than being ported.
Book, Review and User in types.go are now ScrapedBook, ScrapedReview and ScrapedUser, and they go when M4 puts the commands on the extractor.
Doing the port in the same PR would have been M4 done early and badly.

Two things ticked with a note instead of silently.
Conn.Loaded is always 0, because the only connection the book page carries is the work's editions and it carries the URL and nothing else.
full and deep parse and cost correctly but read no further surfaces yet, so either one says so in missed.

--check against the live site, both books exact on the sum and matching to two decimals on the mean:

id       title                fields  buckets sum  ratings   derived  published  ok
2767052  The Hunger Games     40      10232418     10232418  4.3548   4.35       true
1885     Pride and Prejudice  40      4983594      4983594   4.2965   4.30       true

M4 next: every read command onto the extractor, which is also what fills in Loaded, full and deep.

Walked the acceptance list against the code rather than against memory.
Five of the seven were already held by named tests.
Two were not, and both are now merged.

Held already, and by what:

No disallowed request by default: TestNoDisallowedByDefault (goodread/robots_test.go) walks every op in the registry and checks both directions, so an op marked allowed that the rules refuse fails and an op marked disallowed that the rules permit fails too.
TestEverySurfaceHasSampleArgs stops it passing on an empty path, which every rule set permits.
Every v0.2.0 command still there: TestNoV020CommandWasRemoved (cli/policy_test.go) checks the v0.2.0 list by name and by alias, so the quote to quotes rename reads as a rename and not as a removal.
--no-robots not reachable except by typing it: TestNoRobotsNotInConfigFile, TestNoRobotsNotInEnv and TestNoRobotsNotImplied.
The rate floor with the override on: TestPaceFloorHolds sets NoRobots and times five real requests, and TestBurstIsOne and TestNoConcurrency close the two routes around a floor.
No Cookie header: TestCookiesFlagIsGone plus a grep that comes back empty.
Selector debt: the "or" clause. The introduction, the release notes and goodread extraction all say which six surfaces are Rails templates and why a selector is the only rung there.

Not held, now merged:

#30 implements the robots note. A grep for robots_allowed, RobotsAllowed and robots.allowed came back empty across goodread/ and cli/, so this line was simply unimplemented. A record built from a disallowed page now carries a robots block with allowed: false, the path, the rule that matched it and which robots.txt said so. A record off an allowed surface carries nothing, so the absence means as much as the presence. Reviews carry it on the row, because a review has no envelope and reviews --all merges the book page's sample with the walked pages into one list.

Writing the test for it turned up a real bug. The v0.3.0 record fetchers built URLs on the hardcoded BaseURL, so a test that pointed the client at a local server still fetched goodreads.com for real. The first run did exactly that, and the second came back with a WAF challenge, which is how it was noticed. They go through the client's base now.

#31 adds the fetched half of the first line. The offline test reads the capture, and the capture is the thing that goes stale: Goodreads can add a Disallow tomorrow and every offline test here keeps passing while the tool reads a page it is no longer allowed to read. The new test walks every op against the live file, and also compares the capture against it per op so a drift reads as "recapture it". One request at the default pace, skipped under -short, and it skips rather than fails when the site is unreachable.

Run against the live file just now: every op agrees and the capture is current.

So the acceptance list is complete. What is left on the version is the outward-facing half in #17, and that waits on a go-ahead.

Navigation Menu

Repository navigation

v0.3.0: obey robots.txt with an explicit override, and read the Apollo cache instead of the rendered page #8

Why this version exists

M1 robots

M2 extraction

M3 model

M4 commands

M5 store

M6 graph

M7 crawl

M8 mcp

M9 release

Blocked

Acceptance

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

v0.3.0: obey robots.txt with an explicit override, and read the Apollo cache instead of the rendered page #8

Description

Why this version exists

M1 robots

M2 extraction

M3 model

M4 commands

M5 store

M6 graph

M7 crawl

M8 mcp

M9 release

Blocked

Acceptance

Activity

tamnd commented on Aug 15, 2026

tamnd commented on Aug 15, 2026

tamnd commented on Aug 15, 2026

tamnd commented on Aug 15, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions