Skip to content

Make crawls info an operation so -o works - #142

Merged
tamnd merged 1 commit into
mainfrom
fix-crawls-info-renderer
Aug 12, 2026
Merged

tamnd merged 1 commit into
mainfrom
fix-crawls-info-renderer

Conversation

@tamnd

@tamnd tamnd commented Aug 12, 2026

Copy link
Copy Markdown
Owner

X
Closes #140.

crawls info was registered as an escape hatch that printed its own text, so -o was parsed, accepted, and thrown away. Asking for JSONL got the same key: value block as asking for a table, which is the one thing every other read command does not do.

It was also a duplicate of stats that had drifted. stats counted warc,wat,wet,robotstxt,non200responses and crawls info counted warc,wat,wet,robotstxt,cc-index-table, so the same question got two answers depending on which name you happened to type.

Both names now register as one kit operation over a shared count closure and a single statKinds list. The renderer is back in the path and the two spellings agree by construction rather than by someone remembering to edit both. crawls info keeps the optional positional crawl it has always accepted, so a command in a script keeps working, and gains --kinds. stats keeps reading -c.

Measured live

$ ccrawl crawls info -o jsonl
{"crawl":"CC-MAIN-2026-30","kind":"warc","files":100000}
{"crawl":"CC-MAIN-2026-30","kind":"wat","files":100000}
{"crawl":"CC-MAIN-2026-30","kind":"wet","files":100000}
{"crawl":"CC-MAIN-2026-30","kind":"robotstxt","files":900}
{"crawl":"CC-MAIN-2026-30","kind":"non200responses","files":900}
{"crawl":"CC-MAIN-2026-30","kind":"cc-index-table","files":900}

$ ccrawl crawls info -o csv
crawl,kind,files
CC-MAIN-2026-30,warc,100000
...

$ ccrawl crawls info -o table
a box-drawn table with CRAWL, KIND, FILES

$ ccrawl crawls info CC-MAIN-2026-25 -o jsonl; echo $?
CC-MAIN-2026-25 rows
0

$ ccrawl crawls info a b; echo $?
Expected at most 1 argument(s), got 2.
2

Done when

  • crawls info -o jsonl writes JSONL, -o csv writes CSV, -o table writes a table
  • the kinds it counts and the kinds stats counts are the same list, or the difference is written down
  • the positional crawl keeps working
  • the reference shows the new output

Checks

gofmt -l . clean, go vet ./... clean, go test ./... passing, scripts/docs-drift.sh matches the binary at 407 flag mentions, scripts/coverage.sh above every floor with cli at 52.9%, golangci-lint run ./... 0 issues.

New tests: TestCrawlsInfoRendersInEveryFormat unmarshals every JSONL line and checks the CSV header, and TestCrawlsInfoAndStatsCountTheSameKinds compares the kind column of both spellings so the lists cannot drift apart again.

crawls info was an escape hatch that printed its own text, which meant
-o was accepted and ignored: asking for jsonl or csv got the same
key: value block either way. It was also a worse duplicate of stats,
counting five kinds where stats counted five different ones, so the
same question got two answers depending on which name you typed.

Both names now register as the same operation over one count closure
and one statKinds list, so the renderer is back in the path and the
kinds match by construction. crawls info keeps the optional positional
crawl it has always taken and gains --kinds; stats keeps reading -c.
A kind whose manifest cannot be fetched still comes back as a row with
files of -1 rather than being dropped.
@tamnd tamnd added this to the v0.10.1 milestone Aug 12, 2026
@tamnd tamnd added type: bug Something is wrong area: cli Command surface, flags, config labels Aug 12, 2026
@tamnd
tamnd merged commit c2007e0 into main Aug 12, 2026
8 checks passed
@tamnd
tamnd deleted the fix-crawls-info-renderer branch August 12, 2026 22:52
@github-actions github-actions Bot mentioned this pull request Aug 12, 2026
34 tasks done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: cli Command surface, flags, config type: bug Something is wrong

Projects

None yet

Development

Successfully merging this pull request may close these issues.

crawls info ignores -o and prints text whatever you ask for

1 participant