Session Handoff [log-review]: Periodic 404 Log Review, Monthly From 2026-11-15 #474

Next steps, in priority order

1. Run the first monthly 404 pass on or after 2026-11-15. OPERATIONS.md "Logs and Debugging" now carries the whole procedure. Run it from the repository rather than rebuilding it by hand.
   • Use the window block in "Reading a 404 list without being fooled by it" with start='2026-10-15T21:03', where this pass ended. It runs checks/log-review.jq over the joined window.
   • Cross-check the 403s with "A 403 is CrowdSec's until shown otherwise", and check any crawler 403 or 404 by forward-confirmed DNS.
   • Run "Checking that search engines can index both sites". viljoen.family/robots.txt and /sitemap.xml should now answer 200, and a 404 on either is a regression. So is a 404 on /tag/review, /viljoen-family or /20180811_172609507_ios after 2026-10-15 23:04 UTC.
   • When a new scanner shape crowds the list, add it to checks/log-review.jq in the same change that records the pass.
   • Done for a pass means a comment on this issue with the date range, the count left after filtering, and each finding's disposition.
2. After that pass, close this lane out if it found nothing new. Otherwise write the next link with the next monthly date.

External blockers

• Production log reads and CrowdSec reads need the maintainer's go-ahead. Ask once per session in a dialog before the first SSH or archive read. A past session's approval does not carry over.
• Search Console is the maintainer's. Google did not fetch the blog's sitemap.xml between 2026-10-10 and 2026-10-15, while Bing did. Whether the sitemap is submitted in Google Search Console, and the new family sitemap too, is visible only there.
• The cleartext redirect is temporary (302/307) on both sites. That is host-side, filed as ptr727/HomeAutomation-Config#1077. The HomeAutomation-Config session acknowledged it and left it in that backlog.
• The home-network CrowdSec allowlist is still ptr727/HomeAutomation-Config#893. Until it lands, keep production probes to a handful per pass.

Internal dependencies

• None. Each pass stands alone.

State

• main is at 7f9f172 (Promote Slashless Redirects, the Family Sitemap, and 404 Pass Tooling #473), and develop is at 00d0c4a (Commit the 404 Pass Filter and Document the CrowdSec and Crawler Checks #472). Release 1.0.244 is published.
• 1.0.244 is deployed to VPS production by run 38820971701 as release 20261015-230352-38820971701-1. Its sampled live check passed. The local production mirror serves 7f9f172 and passed all 1,293 URLs. VPS staging was not redeployed. Re-read each site's X-Blog-Release rather than trusting this line.
• The local staging mirror serves a test build of Redirect Slashless Map Keys and Give the Family Site a Sitemap #471's branch, which matches develop's redirect content.
• No feature branches or worktrees are left. The primary checkout is on develop and clean.
• ptr727/HomeAutomation-Config#893, #894 and #1077 are open, and all three belong to the HomeAutomation-Config session.

The parked decision queue

Zero issues carry decision. This round asked five questions as it went, all answered: the production-read go-ahead, the slashless fix, the process-in-git shape, the host-side issue, and the merges and release.

What the last round did

• The catch-up pass covered 2026-10-10 22:39 UTC through 2026-10-15 21:03 UTC and is recorded in a comment on this issue. The 2026-10-11 daily pass had been missed.
• #471 redirects slashless slugs.map/terms.map keys through a second map lookup (@mapped_slash, contract now 955 URLs). It also adds the family robots.txt and sitemap.xml.
• #472 commits checks/log-review.jq. It also documents the window join, the CrowdSec cross-check, forward-confirmed crawler DNS, and the indexability checklist in OPERATIONS.md.
• #473 promoted both. Released as 1.0.244 and deployed to production and the local production mirror.
• Filed ptr727/HomeAutomation-Config#1077 for the permanent redirect and told that session.

What not to repeat

• Filtering by a broad extension or root word without checking what the sites serve. The first filter dropped /en/, /index.json, /site.js, /apple-touch-icon.png and legacy .zip uploads. Run every built file and contract URL through the filter as a synthetic 404 after changing it. None may be dropped.
• set -e inside a { } group on the left of a pipe. Bash ignores it there, so a failed find or zcat still yields a partial window. Run producers as plain commands inside a ( set -euo pipefail ... ) subshell, as OPERATIONS.md now does.
• Reading an empty 403 as anything but CrowdSec before matching it. 389 of 395 blocked clients matched CrowdSec's alerts or decisions. The rest received only 403s.
• Trusting a crawler user agent. All six crawler-UA 403s came from cloud VMs. Confirm by reverse DNS, then forward.
• Everything in Session Handoff [log-review]: Periodic 404 Log Review, Daily Through 2026-10-11 Then Monthly #461's and Session Handoff [log-review]: Periodic 404 Log Review, Daily Through 2026-10-11 Then Monthly #455's "What not to repeat" still applies.

New learnings

• A routerless https 404 on a served host can be the edge losing its routes while the Pangolin API restarts. On 2026-10-15 at 09:14 UTC the API answered 502 for about 18 seconds. Host-side and transient.
• The repeatable log-review steps are now in OPERATIONS.md "Logs and Debugging" and checks/log-review.jq, per the maintainer's direction that repeatable process belongs in git rather than in notes.