Skip to content

parse wet and wat write Go field names in JSON #125

Closed

Description

@tamnd

ccrawl parse wet -o jsonl and ccrawl parse wat -o jsonl write Go field names, not the field names every other output uses. WARC in the same command writes snake_case, the table columns are lowercase, and the parquet schemas are snake_case, so the JSON is the odd one out.

Reproduced on real data from CC-MAIN-2025-05:

== wet csv ==         url,language,length,text
== wet jsonl keys ==  RecordID CrawlID URL Date ContentLanguage Text
== wat csv ==         url,status,title,links,content_type
== wat jsonl keys ==  RecordID CrawlID URL Date HTTPStatus ContentType Title Links LinksCount Metas WARCFile WARCOffset WARCLength
== warc jsonl keys == content_length date mime payload_digest record_id status type url

The cause is that pkg/wet.Record and pkg/wat.Record carry no json: tags, so encoding/json falls back to the Go field name. cli/rows.go and cli/convert.go pass those structs straight to the encoder, while the WARC path builds a snake_case map by hand. The same structs are what the HTTP API and the MCP surface return, so those speak Go field names too.

Two things already work around it:

  • cli/index_cmd.go carries a second ContentLanguage field on indexDoc with a comment explaining that a hand written file says language while ccrawl parse wet -o jsonl writes the Go field name.
  • docs/content/reference/cli.md:832 tells the reader that index build --input takes "the shape ccrawl parse writes for a WET file", which is not true of the language field.

The fix is to tag both records with the names the parquet schemas in ccrawl/parquet.go already use, keep reading the old key in index build so a JSONL file written by an older ccrawl still loads, and make the doc line true.

This changes the JSON key names on the parse, convert, api, and serve output for WAT and WET. It goes in a 0.x release with a note in the release notes.

Done when

  • pkg/wet.Record and pkg/wat.Record have json: tags matching the parquet column names
  • ccrawl parse wet -o jsonl and ccrawl parse wat -o jsonl write the same names on real CC-MAIN-2025-05 data
  • ccrawl convert writes the same names for the same records
  • index build --input still reads a file written by an older ccrawl
  • docs/content/reference/cli.md describes the shape that is written
  • a test fails if a field on either record loses its tag

Activity

  1. added this to the v0.10.1 milestone on Aug 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area: cliCommand surface, flags, configarea: warcWARC read and writepriority: P3Durability and polishtype: bugSomething is wrong

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions