Repository navigation
M2: read __NEXT_DATA__ and the Apollo cache instead of the rendered page #10
Description
Every Goodreads page ships its whole Apollo GraphQL normalised cache inline, 129 typed entities on the measured book page.
parse.go is 1,033 lines of CSS selectors reading the rendered output of that same data, and it misses fields that are never rendered at all: ratingsCountDist, textReviewsLanguageCounts, places, characters.
Spec: ~/notes/Spec/3008/02_extraction.md.
Two things that look like details and are not.
Cache keys split on the first colon only, because the id contains ://.
Field keys carry arguments, description({"stripped":true}) and quotes({"pagination":{"limit":1}}), so they get parsed rather than string matched, and the limit read out of the key is what generates an honest missed sentence.
Level 2 gets built and tested even while level 1 answers everything, because it is a cross check today and the thing that survives a framework removal tomorrow.
Checklist
-
nextdata.go, matched on the id attribute rather than on position -
apollo.gowithParseFieldKey, name up to the first paren then the argument JSON - cache key split on the first colon only, since ids contain
:// -
__refresolution, depth limited, visited set, cycles resolve to a stub - unresolved refs become
Ref{Resolved: false}rather than an error - level 2, JSON-LD and
og:, implemented and tested even while level 1 answers everything - a test asserting level 1 and level 2 agree on title, author, image, ISBN, rating value and rating count
- level 3 selectors registered in
selectorFieldswith aSincefield, never inlined -
goodread extractionprints the ladder report -
Extractor.setrecords the level for every field, and a lint test fails on direct assignment -
Extra map[string]json.RawMessageon every entity, unknown fields kept and counted. Moved to M3, since there are no typed entities to hang it off untiltypes.gois rewritten. Landed in M3: the v0.3.0 record model #20 -
goodread verify --sample Nreports unknown and missing fields. Moved to M3 for the same reason.goodread verifyover the pinned captures is in. Landed in M3: the v0.3.0 record model #20 -
goodread verify --strictin CI against pinned captures -
testdata/pages/withcapture.txtas the ledger, andTestCaptureLedger - captures stored gzipped and opened through
gzip.Reader - captures, seven of the eleven: two book pages, prolific author, work editions, work quotes, list, genre. The 404 one cannot be captured, a missing book id returns 202 with an empty body and
x-amzn-waf-action: challenge. The remaining three are single rating, no ISBN and fractional series position, and they wait on finding real examples - lazy decode,
map[string]json.RawMessagefirst, entities on demand
Part of #8.
Activity
tamnd commentedon Aug 15, 2026
Done, in #19. Two items moved to M3 and one cannot be done, all marked above.
The spec was wrong about the ladder, and it matters.
Only the book page is a Next.js route. Author, series, list, genre, editions and quotes are still Rails templates with no Apollo cache, and four of them carry no
og:tags either. So the ladder is per surface, not universal, and for those surfaces level 3 is not debt to be driven to zero, it is the only source there is.goodread extractionsays that rather than printing a zero that reads as good news.TestNoNextDataOnRailsPagespins it, so if Goodreads does move them we find out from a failing test and go and delete selectors.Cache key ids come in three shapes, not one.
The spec said
kca://uris. The measured page has 69 of those, 29 bare integers onUser, and 30 whole JSON objects onShelving, and those objects carrykca://uris several levels down. Splitting on the first colon handles all three, but the test now asserts the round trip rather than the prefix, because that is the invariant that actually holds.You cannot pick the entity by type.
The measured page has three
Bookentities and two are stubs holding nothing but__typenameandid, because something on the page referenced them.Apollo.Rootgoes throughROOT_QUERY, which names the query the page actually ran.On the visited set.
It unmarks on the way out. Only an ancestor being revisited is a cycle. Two fields pointing at the same author is a shared author, and a resolver that keeps the mark drops the second one silently.
The 404 capture.
A book page for a missing id returns 202 with an empty body and
x-amzn-waf-action: challenge, not a 404. There is nothing to capture. The client already reports that as blocked rather than trying to solve it, which is where I want it to stay.Numbers.
Freshly fetched live page: 32 fields at level 1, one at level 2, none on selectors. The ratings distribution sums to the published count and its weighted mean matches the published average to two decimals, which is the check that catches a reversed slice.