Redirect the Old News Sitemap to sitemap.xml

OPERATIONS.md

@@ -129,7 +129,7 @@ The deploy root and the base URL are the only host-specific values. A local run
 
 **A VPS release is named by its deploy run, never by a commit.** The hub's deploy task names each release `<UTC yyyymmdd-HHMMSS>-<run id>-<run attempt>` and verifies the site against that id itself. To check a VPS site by hand afterwards, pass that id as `EXPECT_RELEASE`. Read it from the **Deploy site job**'s log, where `make-release.sh` prints `==> installing release <id>` with the id ending in the run's own id and attempt. The Validate sources job prints the same line earlier in the run, for a scratch bundle stamped with a bare timestamp, so the first match of a log search is the wrong id. Never copy it from the site's `X-Blog-Release` header, since a stale config still serves the previous id and the check would then pass against it. A short SHA never matches, so the check waits out its timeout and fails.
 
-**Production's deploy checks a sample of the contract, and every other run checks all of it.** The production edge bans a client that requests more than 40 distinct pages in a burst. It also bans one when more than 10 distinct pages answer `404`, `403` or `400`. A full run from the GitHub runner therefore gets it banned. The deploy hook sets `SAMPLE_CONTRACT=1` for `production` alone, and the check then requests one URL from each class. That covers each redirect class, each shape of page that must render, each media tree, and the family pages. It also covers `/all/` and `/feed.xml`, which only a redirect reaches. The script holds that to 30 distinct pages at most, and none is expected to answer an error. Staging and the local mirrors keep the whole contract, which is where it is proven. **A hand run against the VPS production site sets `SAMPLE_CONTRACT=1` too**, for the same reason the hook does. The local production mirror sits behind no such edge and keeps the whole contract. `checks/check-live-urls.sh --print-sample` lists what a sampled run requests without requesting anything. [`deploy/README.md`](./deploy/README.md#environment-variables) carries the budget.
+**Production's deploy checks a sample of the contract, and every other run checks all of it.** The production edge bans a client that requests more than 40 distinct pages in a burst. It also bans one when more than 10 distinct pages answer `404`, `403` or `400`. A full run from the GitHub runner therefore gets it banned. The deploy hook sets `SAMPLE_CONTRACT=1` for `production` alone, and the check then requests one URL from each class. That covers each redirect class, each shape of page that must render, each media tree, and the family pages. It also covers `/all/`, `/feed.xml`, and `/sitemap.xml`, the fixed redirect destinations no list names. The script holds that to 30 distinct pages at most, and none is expected to answer an error. Staging and the local mirrors keep the whole contract, which is where it is proven. **A hand run against the VPS production site sets `SAMPLE_CONTRACT=1` too**, for the same reason the hook does. The local production mirror sits behind no such edge and keeps the whole contract. `checks/check-live-urls.sh --print-sample` lists what a sampled run requests without requesting anything. [`deploy/README.md`](./deploy/README.md#environment-variables) carries the budget.
 
 **Always set `SITE_BASE_URL` for anything that is not production.** The base URL is baked into the canonical tag, the feed links, and every absolute permalink, so a mirror built without it serves pages that all point back at the production address. Nothing downstream catches this, because the pages render at the right paths and the build gate passes. `make-release.sh` bridges it to Hugo's own `HUGO_BASEURL` internally, and the effective value is printed on every build for that reason.
 
@@ -288,6 +288,8 @@ A request crosses the proxy before it reaches the site, so no single log answers
 
 **When `ServiceName` is absent the edge answered, and `entryPointName` with `RequestScheme` say why**, which is the difference between a finding and a fault. The common cause is a cleartext request to the TLS port: the `websecure` router carries `tls` and therefore matches TLS requests only, so a plaintext request to 443 matches nothing and Traefik answers its own 404 with a body of a few dozen bytes. That is correct behavior rather than a gap, and it needs no action. Read the two fields together before treating a routerless 404 as a routing problem, because the shape that does deserve investigation is a routerless 404 arriving over **https**, which means a hostname the proxy serves no route for.
 
+**The one routerless https `404` that is not a routing fault is the edge restarting.** It arrives on a hostname the proxy does serve, in a minute that also carries `502`s. The host side restarts the edge nightly and when it pushes a change during the day, and its documentation holds that schedule. The routes return within a minute. Such a window is an intentional update blip, so leave it alone. Investigate only a window that does not look like one, such as one that outlasts a restart or recurs with no `502` beside it.
+
 Two properties of the Caddy side are worth knowing before parsing it. Its access log is `format console`, so each line is a timestamp, a level, and a logger name followed by a JSON object rather than being JSON itself, and a parser that assumes one object per line reads nothing. And `trusted_proxies` is what makes `client_ip` the reader rather than the proxy, which is the same setting "Serving" describes as a security boundary. Without it every request in the log appears to come from one internal address, and the inward pass cannot distinguish a reader from a health check.
 
 ### Reading a 404 list without being fooled by it

checks/README.md

@@ -24,7 +24,7 @@ The contract is enforced by two gates, because one cannot cover both halves:
 | [`check-url-parity.py`][check-url-parity] | Every URL that must render exists as a built page, every legacy image URL resolves, every local asset reference points at a real file, and the count of carried media linked from no page still equals its recorded baseline | Against `public/`, in CI and before any release is installed |
 | [`check-live-urls.sh`][check-live-urls] | Every redirect resolves, and its destination answers | Against a running server, which is the only thing that exercises a redirect |
 
-**Production's deploy checks a sample of the contract rather than all of it.** Its edge bans a client that requests more than 40 distinct pages in a burst, and the full contract is over 1,200. So the deploy hook sets `SAMPLE_CONTRACT=1` there, and `check-live-urls.sh` then requests the first URL of each class in file order. That covers every redirect class in [`deploy/README.md`][deploy-readme-redirects], every shape of page that must render, each media tree, and the family site. It also requests `/all/` and `/feed.xml`, which only a redirect reaches. A sampled redirect is checked for its status and its `Location` and is not followed. The script asserts the sample's budget before its first request, and `checks/tests/test_check_live_urls_sample.py` holds the classes to the table's sizes and the sample to that budget. `checks/check-live-urls.sh --print-sample` prints the sample and its budget without requesting anything. Staging, the local mirrors, and any run without the variable check every URL, which is where the contract itself is proven.
+**Production's deploy checks a sample of the contract rather than all of it.** Its edge bans a client that requests more than 40 distinct pages in a burst, and the full contract is over 1,200. So the deploy hook sets `SAMPLE_CONTRACT=1` there, and `check-live-urls.sh` then requests the first URL of each class in file order. That covers every redirect class in [`deploy/README.md`][deploy-readme-redirects], every shape of page that must render, each media tree, and the family site. It also requests `/all/`, `/feed.xml`, and `/sitemap.xml`, the fixed redirect destinations, since no list names them and a sampled redirect is not followed to one. A sampled redirect is checked for its status and its `Location` and is not followed. The script asserts the sample's budget before its first request, and `checks/tests/test_check_live_urls_sample.py` holds the classes to the table's sizes and the sample to that budget. `checks/check-live-urls.sh --print-sample` prints the sample and its budget without requesting anything. Staging, the local mirrors, and any run without the variable check every URL, which is where the contract itself is proven.
 
 ## The two lists
 
@@ -45,7 +45,7 @@ The contract is enforced by two gates, because one cannot cover both halves:
 
 The 180 tag archives are exactly the tags the migrated posts carry. Three further terms answer with an empty page and are redirects rather than renders: `brultech` and `phyn`, which no published post uses, and `review`, an empty tag that is also a 12-post category. Hugo generates a term page only where posts exist, which is why the render list is 180 tags rather than 183.
 
-**`redirect-urls.txt`, 955 URLs that must resolve but need not render.** These have no Hugo equivalent. Reproducing them would be absurd, and 404ing them discards real inbound links.
+**`redirect-urls.txt`, 956 URLs that must resolve but need not render.** These have no Hugo equivalent. Reproducing them would be absurd, and 404ing them discards real inbound links.
 
 | Shape | Count | Disposition |
 | --- | --- | --- |
@@ -59,6 +59,7 @@ The 180 tag archives are exactly the tags the migrated posts carry. Three furthe
 | Blogger monthly archives | 21 | Redirect to `/all/` |
 | Author archive and pagination | 12 | Redirect to `/`, a single-author blog duplicating home |
 | Blogger feed | 3 | `/feeds/posts/default` and its `/atom.xml` alias, redirected to `/feed.xml` |
+| News sitemap | 1 | `/news-sitemap.xml`, the old platform's news sitemap, redirected to `/sitemap.xml` |
 | Blogger per-post comment feeds | 27 | `/feeds/<post id>/comments/default` to the post, via `blogger.map` |
 | Blogger static pages | 2 | `/p/<slug>.html` to `/<slug>/` |
 | Root files | 2 | `/robots.txt/` to the file and `/osd.xml/` to `/`, via `slugs.map`, as below |

checks/check-live-urls.sh

@@ -61,7 +61,7 @@ SAMPLE_PER_CLASS=1
 SAMPLE_MAX_PATHS=30
 SAMPLE_MAX_STATICS=8
 # Pages only a redirect reaches, so no list names them and a sampled redirect never requests them.
-SAMPLE_DESTINATIONS=(/all/ /feed.xml)
+SAMPLE_DESTINATIONS=(/all/ /feed.xml /sitemap.xml)
 
 # Each classifier sets CLASS rather than printing it, since a subshell per URL costs seconds across the redirect list.
 # The redirect classes are the rows of deploy/README.md's table, with the map shapes last, since a bare .html test would claim two Blogger rows.
@@ -87,6 +87,8 @@ redirect_class() {
 		CLASS='@site_feed'
 	elif [[ $path =~ ^/(feeds/posts/default/?|atom\.xml)$ ]]; then
 		CLASS='@blogger_feed'
+	elif [[ $path == /news-sitemap.xml ]]; then
+		CLASS='@news_sitemap'
 	elif [[ $path =~ $blogger_archive ]]; then
 		CLASS='@blogger_archive'
 	elif [[ $path =~ $blogger_page ]]; then

checks/redirect-urls.txt

@@ -758,6 +758,7 @@
 /img_9377/
 /jellyfish-light-arrow-500x182-1/
 /jellyfish-lighting-ports-500x214-1/
+/news-sitemap.xml
 /osd.xml/
 /p/about.html
 /p/viljoen-family.html

checks/tests/test_check_live_urls_sample.py

@@ -88,7 +88,7 @@ def test_every_redirect_class_in_the_table_is_sampled_with_its_size(self) -> Non
             if kind == "redirect"
         )
         expected = readme_classes()
-        self.assertEqual(len(expected), 16)
+        self.assertEqual(len(expected), 17)
         self.assertEqual(sampled, expected)
 
     def test_render_media_and_family_are_each_sampled(self) -> None:

deploy/Caddyfile

@@ -98,6 +98,9 @@
 		@blogger_feed path /feeds/posts/default /feeds/posts/default/ /atom.xml
 		redir @blogger_feed /feed.xml 301
 
+		@news_sitemap path /news-sitemap.xml
+		redir @news_sitemap /sitemap.xml 301
+
 		# Every monthly archive path answers, including dates the blog never covered.
 		# The archive index is where a visitor can actually find the month.
 		@blogger_archive path_regexp ^/\d{4}_\d{2}_01_archive\.html$

deploy/README.md

@@ -170,7 +170,7 @@ more than 40 distinct pages in a burst. It also bans one when more than 10 disti
 contract is over 1,200 URLs. The sample takes one URL from each redirect class in
 [How the redirects are expressed](#how-the-redirects-are-expressed). It adds one from each shape
 of page that must render, one from each media tree, and the three family pages. It also adds
-`/all/` and `/feed.xml`, which only a redirect reaches. Before its
+`/all/`, `/feed.xml`, and `/sitemap.xml`, the fixed redirect destinations no list names. Before its
 first request, the script asserts that this stays within 30 distinct pages across both hosts. An
 edit that grows the sample therefore fails rather than getting the runner banned. None of the pages is
 expected to answer an error. A sampled redirect is checked for its status and a `Location`, and
@@ -345,9 +345,9 @@ It forwards `Host` and `Accept-Language` unchanged.
 Everything the site does not render is the web server's job, and the workload constrains which server can serve it. Two requirements are load-bearing, so a replacement has to meet both:
 
 - **The query string must be matchable.** 110 `?p=<id>` shortlinks redirect on the query alone. A server that matches on the path only would resolve `/?p=123` as `/`, redirect the homepage, and carry the query through to it.
-- **There must be a lookup primitive.** 314 of the 955 resolve through map files rather than patterns, since no rule can derive their destination. The five maps carry 714 entries between them. A linear scan of that many rules per request is the wrong shape.
+- **There must be a lookup primitive.** 314 of the 956 resolve through map files rather than patterns, since no rule can derive their destination. The five maps carry 714 entries between them. A linear scan of that many rules per request is the wrong shape.
 
-The Caddyfile carries **15 `redir` directives**, reading **5 map files** through **4 `map` blocks**. Eleven directives match on a pattern and four resolve through a map lookup. A map serves wherever no pattern can derive the destination from the input.
+The Caddyfile carries **16 `redir` directives**, reading **5 map files** through **4 `map` blocks**. Twelve directives match on a pattern and four resolve through a map lookup. A map serves wherever no pattern can derive the destination from the input.
 
 Directives and URL classes are not one to one, in both directions. `@mapped` is a single directive serving four classes, because their key spaces are disjoint and merging them keeps those four classes to one lookup. `@uploads` is one directive covering a URL set that is gated separately.
 
@@ -370,15 +370,16 @@ Every class below is a legacy shape, closed by the migration, so no count here m
 | `@site_feed` | 3 | `/feed/`, `/comments/feed/`, `/about/feed/` -> `/feed.xml` |
 | `@mapped` via `terms.map` | 3 | the three empty term archives |
 | `@blogger_feed` | 3 | `/feeds/posts/default` and `/atom.xml` -> `/feed.xml`, Blogger's Atom feed and its alias, which WordPress.com kept as a redirect |
+| `@news_sitemap` | 1 | `/news-sitemap.xml` -> `/sitemap.xml`, the old platform's news sitemap, which crawlers still poll |
 | `@mapped_slash` | 3 | a `slugs.map` or `terms.map` key requested without its trailing slash -> that key's destination. The matcher accepts every such key, including keys absent from the list, and leaves out a dotted path |
 | `@blogger_page` | 2 | `/p/<slug>.html` -> `/<slug>/`, Blogger's static-page shape |
 | `@feed_type` | 2 | a WordPress post, attachment, term, author, site, comments, or about feed at `/feed/atom/`, `/feed/rss/`, `/feed/rss2/` or `/feed/rdf/` -> the same feed at `/feed/`, which another row resolves |
 
-**Those sixteen classes account for every line in [`checks/redirect-urls.txt`](../checks/redirect-urls.txt), with nothing in the contract outside the table.** Completeness is the property worth holding, and the list's line count is how to check it. `check-live-urls.sh` classifies the list by the same rows to sample production. A unit test holds each class's count to the size column above, so a new row needs a new class there.
+**Those seventeen classes account for every line in [`checks/redirect-urls.txt`](../checks/redirect-urls.txt), with nothing in the contract outside the table.** Completeness is the property worth holding, and the list's line count is how to check it. `check-live-urls.sh` classifies the list by the same rows to sample production. A unit test holds each class's count to the size column above, so a new row needs a new class there.
 
 `@uploads` is deliberately absent from that table and from the redirect contract. It rewrites `/wp-content/uploads/(.*)` to `/media/$1`, preserving all 778 legacy image URLs, which are gated by `golden-media-legacy.txt` on their own. Counting them here would double-count a set that has its own list.
 
-`@label` is the seventeenth class and is deliberately **not** in the contract. `/search/label/<Label>` was never a redirect. The old platform answered it with a generic search page that returns 200 for a label that never existed. That makes it a soft 404 that looks alive. `labels.map` sends each label to its term archive and defaults anything unmatched to `/all/`, which is a choice rather than a preservation.
+`@label` is a further class and is deliberately **not** in the contract. `/search/label/<Label>` was never a redirect. The old platform answered it with a generic search page that returns 200 for a label that never existed. That makes it a soft 404 that looks alive. `labels.map` sends each label to its term archive and defaults anything unmatched to `/all/`, which is a choice rather than a preservation.
 
 Two orderings are load-bearing. `@post_child_feed` precedes `@post_child` because both match the same shape and the broader one would claim both. No golden URL is five segments under a date, so `@post_child` cannot swallow a page that must render, and `@uploads` rewrites under a prefix no rendered page occupies.