[tamnd]

Closes #10. Part of #8.

What this replaces

parse.go reads rendered HTML with CSS selectors. The page it is reading rendered that HTML from a normalised GraphQL cache it also ships inline, 129 typed entities on the measured book page. So the selectors are reading the output of the data instead of the data, and they miss the fields that are never rendered at all: ratingsCountDist, textReviewsLanguageCounts, places, characters.

What went in

nextdata.go finds the script tag by id rather than by position, since Goodreads ships several inline scripts and their order is not stable, and hands the bytes to encoding/json with nothing evaluated.

apollo.go parses cache keys and field keys.
ldjson.go is level 2, the schema.org block and the og: tags.
extract.go holds the level accounting.
extract_book.go is the book page extractor.
verify.go and goodread verify run it over the pinned captures.

Three things measurement changed

Cache key ids come in three shapes, not one.
The spec said they are kca:// uris. The measured page has 69 of those, 29 bare integers on User, and 30 whole JSON objects on Shelving, and those objects contain kca:// uris several levels down. Splitting on the first colon still handles all three, but the test asserts the round trip rather than the prefix, because that is the invariant that actually holds.

You cannot pick the entity by type.
The measured page carries three Book entities and two of them are stubs holding nothing but __typename and id, because something on the page referenced them. Apollo.Root goes through ROOT_QUERY, which names the query the page ran.

Only the book page is a Next.js route.
This is the big one and the spec got it wrong. Author, series, list, genre, editions and quotes are still Rails templates with no Apollo cache, and four of them carry no og: tags either. So the ladder is per-surface: for those surfaces level 3 is not debt to be driven to zero, it is the only source there is. goodread extraction says that rather than printing a zero that reads as good news, and TestNoNextDataOnRailsPages pins it so that if Goodreads does move them we find out from a failing test and go and delete selectors.

The bits that look like details and are not

Field keys carry their arguments, description({"stripped":true}) and quotes({"pagination":{"limit":1}}), and they are parsed to the matching paren rather than string matched. TestParseFieldKeyOnEveryRealKey runs the parser over every key in the capture, which is the test a strings.Contains implementation fails.

Reading the limit out of the key is what makes the missed sentence honest. The page asks for one quote, so the record says exactly that and names goodread quotes 2792775, with the number and the id both coming from the page rather than from somebody's memory.

Ref resolution is depth bounded with a visited set, since the book/work/bestBook triangle is on every page. The set unmarks on the way out, because two fields pointing at the same author is a shared author, not a loop, and a resolver that treats it as one silently drops the second.

Both descriptions are kept. The raw one is what a renderer wants and the stripped one is what a text pipeline wants, and deriving either from the other is lossy.

Level accounting

Every field write goes through Extractor.set, so the report is generated rather than maintained, and TestNoDirectFieldWrites fails on any write that goes round it.

$ goodread verify -f table
CAPTURE                        SURFACE  FIELDS  L1  L2  L3  NOTE
book_show_2767052.html.gz      s1       33      32  1   0   -
book_show_1885.html.gz         s1       33      32  1   0   -
author_show_1077326.html.gz    s2       0       0   0   0   no extractor for this surface yet

A surface with no extractor yet is a gap in the work, not drift in the site, so it does not fail --strict.

Captures

Seven, gzipped, with capture.txt as the ledger. TestCaptureLedger fails on a file that does not say where it came from, which is the test that caught a hand-edited fixture in arxiv-cli.

One capture the spec asked for is not here. A book page for a missing id returns 202 with an empty body and x-amzn-waf-action: challenge, not a 404, so there is nothing to capture. The client already reports that as blocked rather than trying to solve it.

Cross check

TestLevelsAgreeAcrossTheLadder compares the cache against ld+json on title, author, image, ISBN, page count, format and language, and on rating counts with a one percent tolerance since the two blocks render a moment apart. That is what level 2 is for: when the rungs disagree, one of them is being read wrong.

Verified live

Freshly fetched book page through the real client: 32 fields at level 1, one at level 2, none on selectors, with the ratings distribution summing to the published count and its weighted mean matching the published average to two decimal places. That last check is the one that catches a reversed slice, which no amount of eyeballing a bar chart would.

Gate

gofmt -l . clean, go vet ./... clean, go test ./... ok for both packages. 25 new tests, all against the captures.