Repository navigation
Description
ccrawl parse wet -o jsonl and ccrawl parse wat -o jsonl write Go field names, not the field names every other output uses. WARC in the same command writes snake_case, the table columns are lowercase, and the parquet schemas are snake_case, so the JSON is the odd one out.
Reproduced on real data from CC-MAIN-2025-05:
== wet csv == url,language,length,text
== wet jsonl keys == RecordID CrawlID URL Date ContentLanguage Text
== wat csv == url,status,title,links,content_type
== wat jsonl keys == RecordID CrawlID URL Date HTTPStatus ContentType Title Links LinksCount Metas WARCFile WARCOffset WARCLength
== warc jsonl keys == content_length date mime payload_digest record_id status type url
The cause is that pkg/wet.Record and pkg/wat.Record carry no json: tags, so encoding/json falls back to the Go field name. cli/rows.go and cli/convert.go pass those structs straight to the encoder, while the WARC path builds a snake_case map by hand. The same structs are what the HTTP API and the MCP surface return, so those speak Go field names too.
Two things already work around it:
cli/index_cmd.gocarries a secondContentLanguagefield onindexDocwith a comment explaining that a hand written file sayslanguagewhileccrawl parse wet -o jsonlwrites the Go field name.docs/content/reference/cli.md:832tells the reader thatindex build --inputtakes "the shapeccrawl parsewrites for a WET file", which is not true of the language field.
The fix is to tag both records with the names the parquet schemas in ccrawl/parquet.go already use, keep reading the old key in index build so a JSONL file written by an older ccrawl still loads, and make the doc line true.
This changes the JSON key names on the parse, convert, api, and serve output for WAT and WET. It goes in a 0.x release with a note in the release notes.
Done when
-
pkg/wet.Recordandpkg/wat.Recordhavejson:tags matching the parquet column names -
ccrawl parse wet -o jsonlandccrawl parse wat -o jsonlwrite the same names on real CC-MAIN-2025-05 data -
ccrawl convertwrites the same names for the same records -
index build --inputstill reads a file written by an older ccrawl -
docs/content/reference/cli.mddescribes the shape that is written - a test fails if a field on either record loses its tag
Activity