[tamnd]

Closes #110.

The content signals guide documented a command that does not exist. content quality - was shown reading a JSONL stream and there was no stream form. The output table listed lang, lang_confidence, text_length, link_density, boilerplate_ratio and readability_score, and QualityReport emits none of those. A --warc flag was documented on three subcommands and a --lang flag on quality, and neither was ever registered. Every pipeline on the page failed.

The done-when list asked whether the streaming form is worth having. Two of these are: scoring one URL at a time is a spot check, and filtering a corpus is the reason the signals exist at all. So the form exists now rather than being deleted from the page.

What changed in the binary

All four content commands take - in place of the URL and read a list from stdin. A line is either a bare URL or a JSON object with a url field, which is what search, columnar and crawl fetch already write with -o jsonl, so a query feeds a content command with no glue in between.

$ ccrawl content outlinks https://go.dev/blog/ -o jsonl -n 3 | jq -r .url | ccrawl content quality - -o jsonl
{"url":"https://go.dev/","word_count":833,"title_length":27,"has_main_content":true,"spam_score":0.1,"is_parked":false}
{"url":"https://go.dev/solutions/case-studies","word_count":1415,"title_length":42,"has_main_content":true,"spam_score":0,"is_parked":false}
{"url":"https://go.dev/solutions/use-cases","word_count":131,"title_length":39,"has_main_content":true,"spam_score":0,"is_parked":false}

A URL that cannot be fetched is named on stderr and the rest of the list carries on, because any real list of seeds has a few dead hosts in it and losing the other nine hundred to them helps nobody:

$ ccrawl content quality - -o jsonl < seeds.txt
{"url":"https://go.dev/blog/","word_count":302,...}
{"url":"https://example.com/","word_count":19,...}
fetch https://nx.invalid/: dial tcp: lookup nx.invalid: no such host, skipping it
1 of the 3 URLs on stdin could not be fetched
exit 0

The exit codes follow the rule the last release settled on. A run that scored at least one page exits 0. A run where every URL failed exits 1, because nothing was learned about those pages, they were never seen. Empty stdin exits 3, which is what dropping Single from these ops buys. A single URL that fails still exits 1.

content outlinks now carries the source URL on every row. One page at a time it was redundant; over a stream the rows of a hundred pages land in one file and an edge with no tail is not an edge.

- is refused under serve, api and mcp. The same handler serves all three surfaces, and stdin there is the JSON-RPC session or the operator's terminal, never the client asking for the URL.

What changed in the docs

The guide is rewritten against the binary. Every field table is the fields the command emits, with what each one measures rather than what its name suggests: spam_score is sixteen English sales phrases at a tenth each and capped at 1, is_parked is a regex on a page under 150 words, has_main_content is fifty words. It says outright that these are blunt, that spam_score reads English only, and that a page scoring 0 is not thereby clean.

Every pipeline on the page was run before it was written down. The stored-WARC section now points at ccrawl parse, since the content commands fetch and do not read archives, and says plainly that scoring a stored capture means refetching the page as it is today, which is a different thing.

The CLI reference had content outlinks down as emitting anchor text, which it has never done. Fixed, along with the quality field list, and the - form is documented there too.

While checking what these commands need, the requirements page turned out to claim crawl fetch reads robots.txt first. It only does with --robots; crawl run is the one that checks by default. That page now says which commands check it, which do not, and that a long list on stdin is what crawl run is for.

Tests

cli/content_cmd_test.go stands two pages up on localhost and drives the commands against them, so the single URL path, the stream, the skipped dead URL, the all-failed error, the empty result, the JSON line with a url field and the resolved relative link are covered without touching the network. One test runs a real search against the fake Common Crawl and feeds every row through the stdin parser, because the documented pipeline rests on a contract between two commands rather than on either of them.

cli coverage 50.7 to 52.3 percent, floor raised to 52. scripts/docs-drift.sh passes.

Done when

• every field in the guide is a field the command emits
• every pipeline in the guide runs, and each one was run before it was written down
• the streaming form was worth having, so it exists