Repository navigation
E19: the politeness throttle is per process, so running three pipelines triples the rate #61
Description
Spec: notes/Spec/2115/07-enhancements.md, item E19.
Problem
The throttle lives in HTTPClient, so it is per process. The pipelines are explicitly designed to run at the same time, urls publish and domains publish and markdown export together, which means three times the request rate against a budget nobody is accounting for. Common Crawl is a nonprofit serving free bandwidth, and this is the one place where being sloppy has an external victim.
What to do
- Add a file locked token bucket under the data dir at
~/data/ccrawl/ratelimit.lock, refilled at a configurable global rate via--global-rateorCCRAWL_GLOBAL_RATE. - Every
HTTPClientacquires from it before any request to data.commoncrawl.org or index.commoncrawl.org. - Degrade to per process behaviour if the lock cannot be taken, and log the effective rate at startup either way.
Done when
- three concurrent pipelines on one host stay within 10 percent of
--global-ratecombined
Estimate: 2 days. This is a guardrail for the crawl work, get it in before E3 goes wide.
Activity
tamnd commentedon Aug 12, 2026
Done in #104.
The budget now lives in a lock file at
<data-dir>/ratelimit.lock. Taking a slot means locking the file, reading the time the next slot comes free, pushing it forward by one interval, unlocking, then sleeping until the slot you were handed. The lock is held for sixteen bytes of read and write, so processes queue on the timestamps rather than on the lock.--global-rateandCCRAWL_GLOBAL_RATEset the gap, 0 turns it off. The default is 200ms, the same as the old per process delay, so one process is unchanged and three now split that budget instead of tripling it. Onlyindex.commoncrawl.org,data.commoncrawl.org,commoncrawl.organd thecommoncrawlbucket draw on it. A host where the file cannot be locked warns once and falls back to the per process delay.Two things came out different from the text here, both deliberate and both written down in the code and the docs:
Columnar scans stay exempt, the same call
--ratealready makes. A scan is thousands of few-kilobyte footer reads and a five per second budget would turn a thirty second query into an hour. So--global-ratebounds everything except columnar scans.The effective rate prints under
-vrather than on every run, because a line about rate limiting above the output ofccrawl searchis noise. The half that is not optional, the limiter having degraded to per process, warns regardless of verbosity.Three concurrent pipelines within 10 percent, measured for real. Three pipelines on one host, each walking twenty crawls and fetching the real
warc.paths.gzfromdata.commoncrawl.org, sharing one data dir, sixty requests total:0.2 percent off. The control shows the budget is binding and not a no-op:
The CDX API was refusing connections throughout this work, so the live pipelines fetch path manifests rather than paging the index. Both go through the same throttled client.
In the suite,
TestGlobalLimiterHoldsTheRateAcrossProcessesre-execs the test binary three times against one lock file behind a barrier, 40 slots each at 20ms, and got 49.96 requests per second against a configured 50.00. It also asserts the file counted exactly 120 grants, so nothing bypassed it. It passed on both the ubuntu and macos CI runners.