Repository navigation
Add discover: a breadth-first walk of the Facebook graph - #5
Merged
Merged
Conversation
feed answers one question about one actor. discover chains the commands: from a seed page, profile, group, or post it follows the object's edges, and from each neighbor it follows theirs, hop by hop, streaming one record per node. The anonymous surface is a shallow star, so the graph is small and deliberate. There are five node kinds (page, profile, group, post, comment) and three edges: posts (actor to post), author (post to actor), and comments (post to leaf comment). --follow takes a preset (content, threads, all) or a comma-separated edge list. Two things fall out of reading the pages Facebook serves to crawlers. Preview comments expose a name but no id, so a comment is a leaf and threads needs depth 2 from an actor seed. A feed post is tagged with the actor it came from, so author from a feed post dedups back to the container; author is a real hop only from a post seed, where the URL encodes the owner. The walker sits behind a narrow grapher interface that *Client satisfies, so the whole BFS is tested over an in-memory fake with no network: edge parsing, the follows matrix, the author hop, threads depth, the dedup cycle, the budget and fanout caps, seed-fatal vs deeper-note errors, and comments as leaves. A seed that will not fetch fails the walk; a failure deeper in the walk becomes a one-line note on stderr and the walk carries on. depth, fanout, and -n keep every walk finite. Adds the discovering guide, a discover entry in the CLI reference, and two smoke cases.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
feedanswers one question about one actor.discoverchains the commands: from a seed (page, profile, group, or post) it follows the object's edges, and from each neighbor it follows theirs, breadth first, streaming one record per node.The graph
The anonymous surface is a shallow star, so the graph is small and deliberate. Five node kinds (page, profile, group, post, comment) and three edges:
postsauthorcomments--followtakes a preset (content= posts+author, the default;threads= posts+comments;all) or a comma-separated edge list.Why the edges are shaped this way
Two things fall out of reading the pages Facebook serves to crawlers:
threadsneeds--depth 2from an actor seed (actor to post to comment).authorfrom a feed post dedups back to the container.authoris a real hop only from a post seed, where the URL encodes the owner; one hop further,postsreaches the rest of that author's feed.Bounds and errors
--depth(default 1),--fanout(default 25), and-n(default 500) keep every walk finite. A seed that will not fetch fails the walk; a failure deeper in the walk becomes a one-line note on stderr (silenced by-q) and the walk carries on.Testing
The walker sits behind a narrow
grapherinterface that*Clientsatisfies, so the whole BFS is tested over an in-memory fake with no network: edge parsing and the unknown-token error, thefollowsmatrix, posts-from-actor, the author hop from a post seed, threads depth, the dedup cycle, the budget and fanout caps, depth-0 seeds-only, seed-fatal and unsupported-kind seeds, deeper feed/comment errors degrading to notes, multiple seeds, and comments as leaves.Also
discovercommand (aliaseswalk,graph) streaming a typedNode(payload + depth/via/parent), reusing each object's own row renderer.discoverentry in the CLI reference, and cross-links from search and datasets.discoversmoke cases.Build, vet, and tests are green; gofmt clean;
go mod tidyis a no-op.