Repository navigation
Keep the VPS Staging Deploy Check From Racing the CrowdSec Crawl Ban #510
Description
Summary
The VPS staging deploy runs the whole URL contract from a GitHub-hosted runner, well over a thousand distinct pages in about half a minute. That trips the edge's CrowdSec crowdsecurity/http-crawl-non_statics scenario on every run. Whether the run passes depends on whether the bouncer picks up the ban decision before the check finishes.
- On 2026-10-24 the 16:19Z staging deploy raised the alert about 6 seconds into a 24-second check and still passed, because the decision had not reached the bouncer yet.
- The 22:50Z staging deploy (release 1.0.263) raised it about 10 seconds into a 34-second check. Every request after that answered an empty
403, and the run failed with 426 findings. The URLs before the ban all passed, security headers included.
The race predates the header assertions. A run that takes a few seconds longer only makes losing it likelier.
What to decide
- Exempt the deploy check on the host side. A CrowdSec allowlist for VPS staging traffic, or for the runner, is owned with the edge configuration, not here. The
X-Blog-Checktag cannot be the key, since it is forgeable and documented as never reaching rate limiting. - Sample VPS staging too. Set
SAMPLE_CONTRACT=1for VPS staging as production already does, and leave the whole contract to the local mirrors, which sit behind no such edge. That changes where OPERATIONS.md "Deploying" says the contract is proven.
Done
A VPS staging deploy no longer fails on a crawl ban, by either route, and OPERATIONS.md says which.
🤖 Generated with Claude Code
Activity
ptr727 commentedon Oct 24, 2026
A re-run of the 1.0.263 staging deploy at 23:26Z lost the race too: 88 findings, every one an empty 403, both 404 probes included. Its check took 41 seconds, against 24 at 16:19Z before the header assertions and 34 on the first failed run. A longer check gives the bouncer more time to pick the ban up, so trimming the check's run time narrows the race but does not close it. Production stays on 1.0.260 until this is settled.
ptr727 commentedon Oct 24, 2026
Deferred by the maintainer at the session close on 2026-10-24: this is a decision for the host side and the VPS deployment to make together, with Blog as one party. The maintainer added that the site should not get trapped by CrowdSec where Cloudflare already covers the same protection. Filed as ptr727/HomeAutomation-Config#1374, and the HomeAutomation-Config session was asked to weigh in there. This keeps the decision label until that issue settles a route.
ptr727 commentedon Oct 25, 2026
The HomeAutomation-Config session proposed a host-side fix on ptr727/HomeAutomation-Config#1374: a CrowdSec postoverflow whitelist that drops only the http-crawl-non_statics overflow when every event targets a blog hostname. The key is the site crawled, not the client, so nothing a client sends can widen it. Probing, CVE, and sensitive-file scenarios stay live. Blog answered with its technical view: it prefers that route over sampling VPS staging, needs no Blog change, and the blog names are blog.insanegenius.com and blog.vps.insanegenius.net. A passing full run returns only the two 404 probes as 4xx. The route still waits on the maintainer's go-ahead on #1374.
ptr727 commentedon Oct 25, 2026
The route is settled: the maintainer approved the host-side exemption on ptr727/HomeAutomation-Config#1374. A CrowdSec whitelist drops only the
http-crawl-non_staticsban when every request targets a blog-stack name, and it covers all four names, the family ones included. Other scenarios still ban on those names. No Blog change is needed.The whitelist is not live yet. It still needs merging and installing on the VPS, behind the maintainer's go-ahead, and until then a staging re-run still loses the race. HomeAutomation-Config will say when it's installed. Then re-run the 1.0.263 VPS staging deploy, close this issue on the passing run, and deploy production.