[tamnd]

Closes #49.

markdown export hardcoded h2m with trafilatura in FavorRecall mode. Which extractor you use is a corpus quality decision, not a build time constant, so it is a flag now.

What changed

--extractor h2m|readability|raw|wet on markdown export, and the three WARC engines on markdown refetch. The readability engine was already written and stranded in markdown_bench_test.go where only the benchmark could reach it, so it moved into ccrawl/extractor.go next to the others. raw renders the whole document with no extraction at all, which is the control corpus: every extractor is a lossy judgement about what a page was for, and the output alone never says where that judgement went wrong.

--source-kind warc|wet picks the manifest. It defaults to whatever the extractor needs, so it rarely has to be given, and a pairing that cannot work is a usage error rather than a silently reinterpreted run. A WET file holds no HTML and a WARC holds no pre-extracted text. wet on refetch is rejected too: a live fetch returns HTML, and there is no Common Crawl text to pass through.

Rows carry a new extractor column as name@version, and the dataset card names the engine in its processing steps. The version is the part that matters, since extraction changes between releases and a dataset that only records h2m cannot explain why two shards built months apart disagree about the same page. The WET text has no version of ours to record, so it is stamped with the crawl as wet@CC-MAIN-2026-30.

open-markdown-v3 now has three appended columns (language, language_confidence, extractor) and changes nothing else, so a v2 reader is unaffected.

Done when

• the same shard exported with three extractors gives three parquet files with identical doc_id sets and different markdown
• the dataset card says which engine produced the data

Real data

Shard 0 of CC-MAIN-2026-30, the same 3.5 GB of HTML through each engine on the same machine:

Engine	Rows	Markdown	Parquet	Mean doc	Wall clock
h2m	20861	91.9 MB	46.8 MB	3990 B	1m49s
readability	20666	108.5 MB	51.8 MB	4865 B	1m58s
raw	20806	446.2 MB	328.2 MB	20724 B	2m55s
wet	21165	161.9 MB	80.0 MB	6604 B	15s

20353 documents are in all three WARC engines, 97.6 percent of what h2m produced, and of the documents h2m and readability share only 284 have byte identical text. So the first box holds with one honest caveat worth stating rather than hiding: the doc_id sets are not exactly identical, because an engine that extracts nothing from a page skips it instead of writing an empty row, and the engines do not give up on quite the same pages. Row counts reproduced exactly across two runs of the same shard.

raw produces roughly five times the text of h2m from the same pages, which is a fair measure of how much of a web page is not the page. wet is in a different cost class, and it is also somebody else's extraction decision, boilerplate included, in plain text rather than Markdown.

Fixed on the way past

The refetch flag checks ran after the manifest download, so a rejected flag combination cost a network round trip before it was rejected. They run before anything is fetched now. And the source-kind mismatch error started with a flag name, which the error renderer title-cases into Source-Kind, so it is worded from the extractor instead.

Gates

gofmt -l ., go vet ./..., go test ./..., golangci-lint run (0 issues), ./scripts/docs-drift.sh (reference matches the binary).