Repository navigation
Answer the columnar filter queries without duckdb - #80
Conversation
The columnar subcommands all needed a duckdb binary on PATH, and printing the SQL when it was missing does not answer anybody's question. This adds a native engine that reads the Parquet index over ranged HTTP and answers the projection and aggregation subset itself: urls, locations, count, langs, mimes and schema, with any combination of the filter flags. It works the way a query planner does. Open a part, read the footer, pull the page index only for the columns the filters mention, and drop any page whose min and max rule the filter out. Because the files are sorted by url_surtkey, a --domain or --host filter also gets a prefix predicate on that column, which prunes hardest. What survives is read a column at a time, cheapest column first, so a row group the first filter empties never has the expensive columns touched. Deciding a part holds nothing costs about 200 KiB out of 8.7 MiB. The domain case carries both surtkey prefixes, the apex form ending in ')' and the subdomain form ending in ',', in one OR predicate. A single prefix silently drops every apex capture, which is the sort of wrong answer that never announces itself. --engine gains native alongside duckdb and print, and auto now takes native for anything native can answer, installed duckdb or not, so the same command behaves the same way on every box. Arbitrary SQL through columnar query and columnar sql --run stays duckdb only and always will. Two client changes came out of measuring it. The transport now keeps 64 idle connections per host, because Go's default of two means every worker past the second pays a fresh TLS handshake. And the engine runs without the inter-request delay, which is priced for requests that mean something rather than thousands of few-kilobyte footer reads, and which duckdb does not apply either. Refs #45
errcheck on a deferred Close in the new test and on an Fprintf that came in with the run journal, and De Morgan on the prefix bounds check, which reads better positively anyway. Refs #45
toInt64 only knew float64 and string, which is what duckdb's JSON gives. The native engine hands back the Parquet column's own type, so a warc offset arrived as an int64 and fell through to zero. A location with a zero offset and length is a record nobody can fetch, and nothing about it looks wrong until the fetch comes back empty. Caught by comparing 370 vnexpress.net locations under both engines against CC-MAIN-2026-30, where the filenames and URLs matched and every offset and length did not. Refs #45
The native engine returns a group under "value" and duckdb under the
column's own name, and runBreakdown passed whichever map it got straight
to the sink. So the counts agreed and the JSONL did not: langs printed
{"value":null,"n":240377} natively and {"content_languages":null,"n":240377}
through duckdb. The row is rebuilt under the column name now.
Refs #45
Cross engine equivalence, all three done when boxesRan every subcommand under both engines against three live crawls,
And with duckdb off PATH entirely, against the default
Two bugs the comparison foundNeither was visible from the fixture tests, and both are the reason this run was worth the hours it took.
A breakdown had a different JSON shape per engine. The native engine returns a group under A note on the load this puts on Common CrawlEach case is a 300 part scan under each engine, and running them back to back got this box 403'd twice. The final run paces itself: two workers on the native side, 20s between the two engines, 6 minutes between cases, and a 15 minute cooldown and retry on a throttle. Worth knowing before anybody repeats it. |
Closes #45.
ccrawl columnarneeded a duckdb binary on PATH for every one of its subcommands. Printing the SQL when duckdb was missing was a polite gesture, not an answer. This adds a native engine that reads the Parquet index itself over ranged HTTP and answers the projection and aggregation subset:urls,locations,count,langs,mimesandschema, with any combination of--domain,--host,--tld,--mime,--lang,--status,--path-prefixand--subset.columnar queryandcolumnar sql --runtake arbitrary SQL and stay duckdb only, permanently. An engine that accepts arbitrary SQL is a database, and this is a command line tool. Asking for--engine nativeon one of those is an error rather than a silent fallback, so a script cannot end up quietly running somewhere else.How it reads less
Open a part, read the footer, pull the page index only for the columns the filters actually mention, and drop any page whose minimum and maximum rule the filter out. Because the files are sorted by
url_surtkey, a--domainor--hostfilter also gets a prefix predicate on that column, which is the one that prunes hardest. What survives is read a column at a time, cheapest column first, and each column narrows the set of rows still alive, so a row group the first filter empties never has the expensive columns touched at all. Only then are the output columns read, and only for the rows that matched.Deciding a part holds nothing costs about 200 KiB out of an 8.7 MiB file.
The bug worth calling out
The
--domainpredicate carries both surtkey prefixes in one OR: the apex host's captures end the host with)and every subdomain's continue with,. A single prefix, which is what I wrote first, silently drops every apex capture. That is the kind of wrong answer that never announces itself, soTestNativeDomainMatchesApexAndSubdomainspins it.Three things measurement changed
parquet-go's
OptimisticReadis actively harmful for a ranged reader that already has its own block cache: it speculatively pulls about 1.7 MiB of tail per file. Turning it off, plusSkipMagicBytesand lazy per-columnColumnIndex(), took open plus prune from 4.9 MiB to about 200 KiB per part.The transport now keeps 64 idle connections per host. Go's default is two, which is right for a browser and wrong here, because every command that fans out talks to one host and every worker past the second was paying a fresh TLS handshake.
The 200 ms inter-request delay is the wrong unit for a columnar scan. It is priced for requests that mean something, a CDX page or a WARC range, and a scan is thousands of few-kilobyte reads of footers and column indexes. duckdb, answering the same query, applies no delay at all. The engine now runs on a client without it, which took the benchmark from 2:09 to 15.3s.
Separately, and not fixed here:
--rate 0cannot disable the throttle from the CLI at all, because any-cli v0.4.1resolveConfigdoesif g.rate > 0. That is a dependency bug and wants its own report upstream.Speed
Counting the Vietnamese TLD across the 300 parts of one crawl's
robotstxtsubset, both engines agreeing on 240377:-j 16-j 8The bar in the issue was native being no worse than 2x slower. Being faster is not the point and will not hold for every query, since duckdb is vectorized and will win where a query really does have to read a lot of rows. The point is that the no-dependency path is not a consolation prize.
One deliberate deviation, now reverted to the spec
I first wired
autoto prefer duckdb wherever duckdb was installed, on the theory that a new engine should not change what an existing query runs on. The issue asks for the opposite, and once the two engines were shown to agree and native measured faster, the issue is right.autonow takes native for anything native can answer, installed duckdb or not, so the same command does the same thing on every box.--engine duckdbforces the old path.Testing
ccrawl/columnar_native_test.gobuilds a real multi-row-group Parquet fixture and serves it overhttptestwith real range support, covering counts, the apex plus subdomain domain case, projection and limit, group counts including a NULL group kept distinct from an empty string, schema naming, row group pruning, expressibility and error paths. The fixture uses*stringfor languages because Go's""and a real NULL are different rows and the first version of the test could not tell them apart.Verified against live Common Crawl, not just the fixture, for both the schema (identical to duckdb's, column for column and type for type) and the numbers above.