Milestone: v0.10.1, what the docs promised

Tracking issue for the v0.10.1 milestone. It is a patch release for the things v0.10.0 shipped with, plus the one item v0.10.0 could not measure.

Goal

Close the gap between what a page of documentation promises and what the binary does, and land the CDX filter push once the index server is reachable enough to measure it.

Checklist

Documentation that described something that did not exist:

content-signals guide documents a content quality command that does not exist #110 content-signals guide documents a content quality command that does not exist, fixed by building the streaming form in Make the content commands do what the guide said they did #118
The dataset card points readers at open-markdown-v2, but export publishes v3 #88 the dataset card points readers at open-markdown-v2 while export publishes v3, fixed in Name the repo the card is committed to #119

Messages and checks:

Carried over from v0.10.0, and carried on to v0.10.2:

E16: push search filters to the CDX server #60 E16, push search filters to the CDX server, moved to v0.10.2. PR Send the URL substring filters to the index server #112 is written and green and stays open; both done-when boxes are byte counts against index.commoncrawl.org, which has refused connections for 22 days. The reason is written down on E16: push search filters to the CDX server #60.

Ship gate for v0.10.1

Open question for the maintainer

main has no branch protection, so gh pr merge --auto merges as soon as GitHub considers a PR mergeable, whether or not ci passed. That is how #118 merged with docs-drift red. Making the ci checks required is a repository setting rather than a code change, so it is a call for you.

Process

Every PR that lands against this milestone gets a comment on this issue with the PR number and one line on what changed, and the matching checklist box is ticked when the linked issue closes. Automated by .github/workflows/milestone-tracker.yml. #110 and #88 merged before this issue existed, so their boxes are ticked by hand here.

Merged #118, Make the content commands do what the guide said they did.

Closes:

content-signals guide documents a content quality command that does not exist #110 content-signals guide documents a content quality command that does not exist

The guide promised content quality - reading URLs from stdin, six output fields, a --warc flag and a --lang flag, none of which existed. All four content commands now take - and read a URL per line, a JSON line with a url field works too so search and crawl fetch output pipes straight in, a URL that fails is named and skipped, and the guide was rewritten from what the commands actually emit. Two more real doc errors turned up on the way: the CLI reference claimed content outlinks carries anchor text, and the requirements page claimed crawl fetch reads robots.txt when it only does with --robots.

Merged #119, Name the repo the card is committed to.

Closes:

The dataset card points readers at open-markdown-v2, but export publishes v3 #88 The dataset card points readers at open-markdown-v2, but export publishes v3

The card is now built from the repo the run is pushing to, so a run publishing to somebody else's dataset gets snippets pointing at theirs instead of ours, and local_dir follows the repo name. A card rendered outside a run falls back to ccrawl.DefaultMarkdownRepo, which is also the export flag default now, so the two cannot drift apart again.

Merged #123, Check the requirements page against the crawl commands too.

Closes:

docs-drift has been failing on main since #118 #122 docs-drift has been failing on main since Make the content commands do what the guide said they did #118

Merged #124, Stop printing flags that do not exist.

Closes:

Error messages name flags that do not exist, because the renderer title-cases the first word #121 Error messages name flags that do not exist, because the renderer title-cases the first word

Merged #126, Give the WAT and WET records their JSON names.

Closes:

parse wet and wat write Go field names in JSON #125 parse wet and wat write Go field names in JSON

Merged #128, Stop printing help nobody wrote.

Closes:

Flag help prints words nobody wrote, for the same reason the errors did #127 Flag help prints words nobody wrote, for the same reason the errors did

Merged #130, rank: find the table instead of demanding one.

Closes:

rank host and rank domain demand a URL the program can work out #129 rank host and rank domain demand a URL the program can work out

Merged #132, exit 8 when Common Crawl cannot be reached.

Closes:

A network failure exits 1, and the help says it exits 8 #131 A network failure exits 1, and the help says it exits 8

Merged #136, Refuse crawl seed --max-tier 1 instead of scanning for nothing.

Closes:

crawl seed --max-tier 1 reads the whole rank table and emits nothing #135 crawl seed --max-tier 1 reads the whole rank table and emits nothing

Merged #139, Keep working from a cached crawl list when the index server is down.

Closes:

A cached crawl list is thrown away the moment the index server is down #137 A cached crawl list is thrown away the moment the index server is down

Merged #142, Make crawls info an operation so -o works.

Closes:

crawls info ignores -o and prints text whatever you ask for #140 crawls info ignores -o and prints text whatever you ask for

Merged #143, Make --data-dir move the cache with it.

Closes:

--data-dir does not move the cache, though the reference says it does #138 --data-dir does not move the cache, though the reference says it does

Merged #145, Leave out host counts nothing measured.

Closes:

host top and host get report zero for numbers nothing measured #144 host top and host get report zero for numbers nothing measured

Merged #147, Give a read that fails partway through a response exit 8.

Closes:

A read that fails partway through a response exits 1, not 8 #146 A read that fails partway through a response exits 1, not 8

Merged #148, Refuse an output format nothing can encode.

Closes:

An unknown -o value is accepted and silently writes JSONL #133 An unknown -o value is accepted and silently writes JSONL

Merged #151, Let ccrawl version render as data.

Closes:

ccrawl version ignores -o and prints one prose line #150 ccrawl version ignores -o and prints one prose line

Merged #134, docs: release notes for v0.10.1.

v0.10.1 is out: https://github.com/tamnd/ccrawl-cli/releases/tag/v0.10.1

Sixteen issues closed and seventeen pull requests merged. Every box above is ticked. The release notes are at docs/content/release-notes/v0-10-1.md.

One thing this milestone set out to do is not done. #60 wanted a byte count for the pushed CDX filters, and index.commoncrawl.org has been refusing connections for twenty two days, so the measurement cannot be taken. The change itself is written and green in PR #112, but the whole claim it makes is a number, and there is no honest way to publish a number nobody took. #60 and #112 have moved to v0.10.2 and the release notes say plainly that the promise made in v0.10.0 is not kept yet and why.

Checked against the published darwin arm64 binary rather than a local build:

$ ccrawl version
ccrawl 0.10.1 (commit 7813222, built 2026-08-13T02:47:33Z, darwin/arm64, go1.26.5)

$ ccrawl version -o json | jq -r .commit
7813222

$ ccrawl search 'example.com/*' -o csvv; echo $?
ccrawl: -o csvv: unknown output format, use one of auto, table, markdown, list, json, jsonl, csv, tsv, url, raw, md, section, sections, template, parquet
2

$ ccrawl content quality http://127.0.0.1:9/gone -o jsonl; echo $?
Fetch http://127.0.0.1:9/gone: Get "http://127.0.0.1:9/gone": dial tcp 127.0.0.1:9: connect: connection refused.
8

$ ccrawl content quality 'http://[::1' -o jsonl; echo $?
Fetch http://[::1: parse "http://[::1": missing ']' in host.
1

Closing this one. v0.10.2 is open and has #60 in it.

Navigation Menu

Repository navigation

Milestone: v0.10.1, what the docs promised #120

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Milestone: v0.10.1, what the docs promised #120

Description

Activity

tamnd commented on Aug 12, 2026

tamnd commented on Aug 12, 2026

github-actions commented on Aug 12, 2026

github-actions commented on Aug 12, 2026

github-actions commented on Aug 12, 2026

github-actions commented on Aug 12, 2026

github-actions commented on Aug 12, 2026

github-actions commented on Aug 12, 2026

github-actions commented on Aug 12, 2026

github-actions commented on Aug 12, 2026

github-actions commented on Aug 12, 2026

github-actions commented on Aug 12, 2026

github-actions commented on Aug 12, 2026

github-actions commented on Aug 12, 2026

github-actions commented on Aug 12, 2026

github-actions commented on Aug 13, 2026

github-actions commented on Aug 13, 2026

tamnd commented on Aug 13, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions