Repository navigation
M6: the twelve edges, traversal and export - #25
Merged
Merged
Conversation
The edge set is closed now. 04_graph.md section 4 names twelve predicates and predicates.go holds them as constants with their endpoints, their surfaces and what they carry, and PutEdge refuses anything else. The refusal is at the write, where the name is still in front of somebody, rather than at the read, where a typo looks like missing data. Four of the names the indexer was writing were not the spec's. `wrote` was an author to book edge and is now `contributed_by` running book to author, `contains` was list to book and is now `listed_in` running book to list, and `in_genre` is now `shelved_as`. Directions are the spec's rather than the ones that read best in a sentence, because Edges answers either way and one canonical direction is what stops the same fact being written twice under two names. The properties that ride on an edge are there too. `contributed_by` carries the role, which is the difference between an author and an illustrator, and `in_series` carries the position both raw and parsed, because the raw string is the only one that holds 1-3 on an omnibus and the number is the only one that sorts. `listed_in` carries the rank, which belongs to the edge because a book sits at 3 on one list and 900 on another. Places, characters and awards got tables. They have no id on Goodreads and no page of their own, so the slug is the id and two places with the same name in different countries collide. That limit is in the spec rather than papered over with a synthetic id nothing else in the world shares. `goodread graph <uri>` walks the local store, defaults to depth 1, and asks for --yes above 500 nodes. Cycles are the normal case and not the error case: Book to Work to BestBook closes in three hops on most popular titles, so the walk carries a visited set from the first hop. A node with an edge and no record is still reported, because that is the most useful thing a walk can say to somebody deciding what to crawl next. `goodread export` writes JSONL or Turtle. JSONL is the whole record and not the columns, since the columns are an index over it. The RDF is schema.org where it fits and grv: where it does not, with schema:exampleOfWork for the book to work edge, which is the standard's own edition versus work distinction and lines up exactly with ours. A wrong predicate is worse than a local one.
9 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #14.
The edge set is closed
goodread/predicates.goholds the twelve predicates from04_graph.mdsection 4 as constants, each with its endpoints, the surfaces it comes from and what it carries.PutEdgerefuses anything else. The guard is at the write, where the name is still in front of somebody, rather than at the read, where a typo looks like missing data.Four names the M5 indexer was writing were not the spec's:
wrote, author to bookcontributed_by, book to authorcontains, list to booklisted_in, book to listin_genreshelved_asedition_of,in_seriesDirections are the spec's rather than the ones that read best in a sentence.
Edgesanswers either way, and having one canonical direction is what stops the same fact being stored twice under two names.What rides on an edge
contributed_bycarries the role,in_seriescarries the position raw and parsed,listed_incarries the rank,woncarries the category and the year. The rank is on the edge because a book sits at 3 on one list and 900 on another, and the rank belongs to neither of them alone.best_edition,set_in,features,wonandquoted_fromare written now where the record carries the data. Places, characters and awards got tables: they have no id on Goodreads and no page of their own, so the slug is the id, and two places with the same name in different countries collide. That limit is written down rather than papered over with a synthetic id nothing else in the world shares.Traversal
goodread graph <uri>walks the local store and fetches nothing.--depthdefaults to 1, per section 5.--yes.--edgesprints the relationships instead of the nodes.Export
goodread export, section 6.--to jsonlis the default, one node per line, the whole record and not the columns. The columns are an index over the record and a consumer that got only them would be missing every field this version has no column for.--edgeswrites the edge table instead.--to rdfis Turtle: schema.org where it fits,grv:where it does not.schema:exampleOfWorkfor book to work, which is the standard's own edition versus work distinction and lines up exactly with ours. The aggregate rating goes on a blankschema:AggregateRatingbecause that is where schema.org puts it, and the rating distribution stays agrv:list because nothing in schema.org holds five numbers in rating order.--toand not--format, because--formatis the global flag that picks how records print.Tests
TestTheEdgeSetIsTheSpecsTwelve, exact names, endpoints that are real node kinds, and the two the spec calls out for their properties.TestAnUnknownPredicateIsRefused, and the refusal names what to use instead.TestEveryEdgeHasAnRDFPredicate, so no edge can vanish from the export without a word.TestBothIDSpacesSurviveTheRoundTrip, reading the kca id and the legacy id back out of the columns off a real capture.TestWalkFinishesOnACycle, six deep through a two hop cycle, on a timeout so a hang is a failure and not a hung suite.TestWalkSizeAgreesWithWalk, because a count that disagreed with the walk would gate the wrong walks.TestExportRDFMapsOntoSchemaOrgandTestExportJSONLWritesTheWholeRecord, both off the realbook_show_2767052capture.graphandexport.gofmt -l . && go vet ./... && go test ./...is clean.