Repository navigation
Make crawls info an operation so -o works - #142
Merged
Merged
Conversation
crawls info was an escape hatch that printed its own text, which meant -o was accepted and ignored: asking for jsonl or csv got the same key: value block either way. It was also a worse duplicate of stats, counting five kinds where stats counted five different ones, so the same question got two answers depending on which name you typed. Both names now register as the same operation over one count closure and one statKinds list, so the renderer is back in the path and the kinds match by construction. crawls info keeps the optional positional crawl it has always taken and gains --kinds; stats keeps reading -c. A kind whose manifest cannot be fetched still comes back as a row with files of -1 rather than being dropped.
34 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
X
Closes #140.
crawls infowas registered as an escape hatch that printed its own text, so-owas parsed, accepted, and thrown away. Asking for JSONL got the samekey: valueblock as asking for a table, which is the one thing every other read command does not do.It was also a duplicate of
statsthat had drifted.statscountedwarc,wat,wet,robotstxt,non200responsesandcrawls infocountedwarc,wat,wet,robotstxt,cc-index-table, so the same question got two answers depending on which name you happened to type.Both names now register as one kit operation over a shared
countclosure and a singlestatKindslist. The renderer is back in the path and the two spellings agree by construction rather than by someone remembering to edit both.crawls infokeeps the optional positional crawl it has always accepted, so a command in a script keeps working, and gains--kinds.statskeeps reading-c.Measured live
Done when
crawls info -o jsonlwrites JSONL,-o csvwrites CSV,-o tablewrites a tablestatscounts are the same list, or the difference is written downChecks
gofmt -l .clean,go vet ./...clean,go test ./...passing,scripts/docs-drift.shmatches the binary at 407 flag mentions,scripts/coverage.shabove every floor with cli at 52.9%,golangci-lint run ./...0 issues.New tests:
TestCrawlsInfoRendersInEveryFormatunmarshals every JSONL line and checks the CSV header, andTestCrawlsInfoAndStatsCountTheSameKindscompares the kind column of both spellings so the lists cannot drift apart again.