[3.x] Support pages with more output file extensions (Sitemap inclusion policy)

EPIC_NON_HTML_PAGES.md

@@ -158,6 +158,17 @@ a standalone feature in its own right.
 > agent workflow warns about — the discovered-page tests would pass while the
 > InMemoryPage-backed generated pages regress.
 
+> **Implemented (PR 4):** `HydePage::showInSitemap()` reads the `sitemap` front
+> matter key, defaulting to whether the resolved output path (`getOutputPath()`)
+> ends in `.html`, per the constraint above. Front matter wins in both directions,
+> so `sitemap: true` opts a non-HTML page back in. One refinement to "defaults
+> `false` for redirects": `Redirect` overrides `showInSitemap()` to return `false`
+> unconditionally, mirroring its `showInNavigation()` and the already-recorded
+> release note that redirect routes are intrinsically excluded from navigation and
+> sitemaps — a redirect page has no front matter channel in `hyde.redirects`, and
+> listing redirects in a sitemap is an SEO anti-pattern, so an opt-in would only
+> be a trap.
+
 ### D4: Generators become container-resolved pages; generator actions stay
 
 Each generated file is registered as an `InMemoryPage` whose compiled contents
@@ -322,7 +333,7 @@ would introduce extension-specific framework surface without solving the broader
 verbatim-file problem. Documentation will instead show the service provider /
 `booting()` registration pattern for custom text output.
 
-### PR 4 — Sitemap inclusion policy
+### PR 4 — Sitemap inclusion policy ✅ Implemented
 
 Goal: pages control their own sitemap presence; fix the production `search.json` leak.
 
@@ -334,6 +345,24 @@ Goal: pages control their own sitemap presence; fix the production `search.json`
 - Changelog note: search indexes no longer appear in sitemaps (bugfix).
 - Independent of PRs 1-3; must land before or with PR 5.
 
+Implementation notes (branch `v3/non-html-pages-sitemap-inclusion-policy`):
+
+- Implemented exactly per D3 (see the D3 "Implemented" note for the front matter
+  semantics and the `Redirect` refinement). `showInSitemap()` joined the
+  `BaseHydePageUnitTest` contract; the InMemoryPage unit test covers the
+  identifier-encoded non-HTML default that the static-extension tests cannot.
+- The non-HTML self-exclusion is verified end-to-end: a registered `robots.txt`
+  `InMemoryPage` is built by the real `build` command and asserted absent from
+  the built `sitemap.xml`, guarding the D3 resolved-output-path constraint
+  against regression by construction rather than only at the unit level.
+- Two existing tests asserted the leak as expected behavior and were flipped:
+  `SitemapServiceTest` now asserts the docs search *page* stays while the search
+  *index* is excluded, and the `SitemapFeatureTest` expected XML dropped its
+  `docs/search.json` entry (it also gained a `sitemap: false` page proving the
+  front matter opt-out through the `build:sitemap` command).
+- No UPGRADE.md entry: the fix requires no user action, and nothing realistic
+  depended on search indexes appearing in sitemaps.
+
 ### PR 5 — Convert sitemap and RSS from build tasks to pages
 
 Goal: `sitemap.xml` and `feed.xml` are routes — served by `hyde serve`, listed in

HYDEPHP_V3_PLANNING.md

@@ -24,6 +24,7 @@ Having this document in code lets us know the devlopment state at any given poin
 - Redirects can now be declared as source and destination path pairs in the `hyde.redirects` configuration array. Hyde registers them with the kernel, includes them in `route:list`, and generates them through the normal site build.
 - Added Blade Blocks for rendering Blade and Blade components from fenced code blocks in Markdown pages. The supported directives are `blade render` and `blade component(name)`, and the feature is controlled by `markdown.enable_blade`. ([#2504](https://github.com/hydephp/develop/pull/2504))
 - Pages can now compile to non-HTML output files. Page classes declare their output file extension through the new static `$outputExtension` property (defaulting to `.html`), and in-memory page identifiers can declare a `.json`, `.txt`, or `.xml` extension directly, so `InMemoryPage::make('robots.txt', contents: ...)` compiles to `_site/robots.txt` through the standard site build. Only the HTML extension is implicit in route keys: pages compiled to non-HTML files keep their extension in the route key, formalizing the convention already used by the documentation search index.
+- Pages can now control their own sitemap inclusion. Set `sitemap: false` in a page's front matter to exclude it from the generated `sitemap.xml`, or override the new `HydePage::showInSitemap()` method in custom page classes. Pages compiled to non-HTML output files (like `robots.txt`) are excluded by default, and `sitemap: true` front matter opts such a page back in.
 
 ### Feature Changes
 
@@ -32,6 +33,7 @@ Having this document in code lets us know the devlopment state at any given poin
 
 ### Minor Changes and Cleanup
 
+- Fixed documentation search index files leaking into the generated sitemap: `search.json` (and any other page compiled to a non-HTML output file) no longer appears in `sitemap.xml`. The sitemap generator now asks each page through `HydePage::showInSitemap()` instead of only filtering out redirect pages.
 - The `Redirect` page class constructor now accepts an optional `$matter` parameter, used by the framework to hide the generated documentation root redirect from navigation menus. Existing usages are unaffected.
 - The realtime compiler now resolves registered page routes before proxying static assets, replacing the hardcoded `search.json` exemption, so `hyde serve` serves any registered route regardless of its output extension. Registered pages now always win over a static file at the same path; the previous behavior of serving such a shadowing file only affected the dev server and no real setups are expected to be affected.
 

docs/_data/partials/hyde-pages-api/hyde-page-methods.md

@@ -1,7 +1,7 @@
 <section id="hyde-page-methods">
 
 <!-- Start generated docs for Hyde\Pages\Concerns\HydePage -->
-<!-- Generated by HydePHP DocGen script at 2026-07-13 14:14:16 in 2.50ms -->
+<!-- Generated by HydePHP DocGen script at 2026-07-13 18:08:53 in 4.67ms -->
 
 #### `make()`
 
@@ -275,6 +275,16 @@ Can the page be shown in the navigation menu?
 $page->showInNavigation(): bool
 ```
 
+#### `showInSitemap()`
+
+Can the page be shown in the sitemap?
+
+It can be explicitly set in the front matter using the `sitemap` key, otherwise it defaults to true for pages compiled to HTML files, and false for pages compiled to non-HTML files like `robots.txt`.
+
+```php
+$page->showInSitemap(): bool
+```
+
 #### `navigationMenuPriority()`
 
 Get the priority of the page in the navigation menu.

packages/framework/src/Framework/Features/XmlGenerators/SitemapGenerator.php

@@ -15,7 +15,6 @@
 use Hyde\Facades\Filesystem;
 use Hyde\Pages\InMemoryPage;
 use Hyde\Support\Models\Route;
-use Hyde\Support\Models\Redirect;
 use Illuminate\Support\Carbon;
 use Hyde\Pages\DocumentationPage;
 use Hyde\Foundation\Facades\Routes;
@@ -31,7 +30,7 @@ class SitemapGenerator extends BaseXmlGenerator
     public function generate(): static
     {
         Routes::all()->each(function (Route $route): void {
-            if (! $route->getPage() instanceof Redirect) {
+            if ($route->getPage()->showInSitemap()) {
                 $this->addRoute($route);
             }
         });

packages/framework/src/Pages/Concerns/HydePage.php

@@ -32,6 +32,7 @@
 use function rtrim;
 use function sprintf;
 use function str_contains;
+use function str_ends_with;
 use function str_starts_with;
 
 /**
@@ -408,6 +409,18 @@ public function showInNavigation(): bool
         return ! $this->navigation->hidden;
     }
 
+    /**
+     * Can the page be shown in the sitemap?
+     *
+     * It can be explicitly set in the front matter using the `sitemap` key,
+     * otherwise it defaults to true for pages compiled to HTML files, and
+     * false for pages compiled to non-HTML files like `robots.txt`.
+     */
+    public function showInSitemap(): bool
+    {
+        return filter_var($this->matter('sitemap', str_ends_with($this->getOutputPath(), '.html')), FILTER_VALIDATE_BOOLEAN);
+    }
+
     /**
      * Get the priority of the page in the navigation menu.
      */

packages/framework/src/Support/Models/Redirect.php

@@ -45,6 +45,11 @@ public function showInNavigation(): bool
         return false;
     }
 
+    public function showInSitemap(): bool
+    {
+        return false;
+    }
+
     /**
      * @throws \Hyde\Framework\Exceptions\InvalidConfigurationException If the path ends in a non-HTML output extension,
      *                                                                  as meta refresh redirects only work for HTML pages.

packages/framework/tests/Feature/NonHtmlPageOutputTest.php

@@ -109,6 +109,21 @@ public function testInMemoryPageWithDeclaredExtensionCanCompileUsingView()
         $this->assertSame('User-agent: *', file_get_contents(Hyde::path('_site/robots.txt')));
     }
 
+    public function testBuildCommandExcludesNonHtmlPagesFromTheSitemap()
+    {
+        $this->withSiteUrl();
+
+        Hyde::kernel()->booting(function (HydeKernel $kernel): void {
+            $kernel->pages()->addPage(new InMemoryPage('robots.txt', contents: 'User-agent: *'));
+        });
+
+        $this->artisan('build')->assertExitCode(0);
+
+        $this->assertFileExists(Hyde::path('_site/robots.txt'));
+        $this->assertFileExists(Hyde::path('_site/sitemap.xml'));
+        $this->assertStringNotContainsString('robots.txt', file_get_contents(Hyde::path('_site/sitemap.xml')));
+    }
+
     public function testBuildCommandCompilesDiscoverableCustomPageClassWithNonHtmlOutputExtension()
     {
         $this->directory('_leaves');

packages/framework/tests/Feature/RedirectTest.php

@@ -95,4 +95,9 @@ public function testRedirectsAreHiddenFromNavigation()
     {
         $this->assertFalse((new Redirect('foo', 'bar'))->showInNavigation());
     }
+
+    public function testRedirectsAreHiddenFromSitemaps()
+    {
+        $this->assertFalse((new Redirect('foo', 'bar'))->showInSitemap());
+    }
 }

packages/framework/tests/Feature/Services/SitemapServiceTest.php

@@ -10,8 +10,11 @@
 use Hyde\Facades\Filesystem;
 use Hyde\Framework\Features\XmlGenerators\SitemapGenerator;
 use Hyde\Hyde;
+use Hyde\Pages\InMemoryPage;
+use Hyde\Support\Models\Route;
 use Hyde\Testing\TestCase;
 use Hyde\Foundation\HydeKernel;
+use Hyde\Foundation\Facades\Routes;
 use Illuminate\Support\Facades\File;
 
 #[\PHPUnit\Framework\Attributes\CoversClass(\Hyde\Framework\Features\XmlGenerators\SitemapGenerator::class)]
@@ -94,18 +97,64 @@ public function testGenerateAddsDocumentationPagesToXml()
         $this->restoreDocumentationSearch();
     }
 
-    public function test_generate_adds_documentation_search_pages_to_xml()
+    public function testGenerateAddsDocumentationSearchPageButNotSearchIndexToXml()
     {
         Filesystem::touch('_docs/foo.md');
 
         $service = new SitemapGenerator();
         $service->generate();
 
-        $this->assertCount(5, $service->getXmlElement()->url);
+        $this->assertCount(4, $service->getXmlElement()->url);
+        $this->assertStringContainsString('docs/search.html', $service->getXml());
+        $this->assertStringNotContainsString('search.json', $service->getXml());
 
         Filesystem::unlink('_docs/foo.md');
     }
 
+    public function testGenerateDoesNotAddPagesWithSitemapFrontMatterSetToFalse()
+    {
+        $this->file('_pages/foo.md', "---\nsitemap: false\n---\n\n# Foo");
+
+        $service = new SitemapGenerator();
+        $service->generate();
+
+        $this->assertCount(2, $service->getXmlElement()->url);
+        $this->assertStringNotContainsString('foo', $service->getXml());
+    }
+
+    public function testGenerateDoesNotAddPagesWithSitemapFrontMatterSetToQuotedFalseString()
+    {
+        $this->file('_pages/foo.md', "---\nsitemap: \"false\"\n---\n\n# Foo");
+
+        $service = new SitemapGenerator();
+        $service->generate();
+
+        $this->assertCount(2, $service->getXmlElement()->url);
+        $this->assertStringNotContainsString('foo', $service->getXml());
+    }
+
+    public function testGenerateDoesNotAddPagesWithNonHtmlOutputPaths()
+    {
+        Routes::addRoute(new Route(new InMemoryPage('robots.txt')));
+
+        $service = new SitemapGenerator();
+        $service->generate();
+
+        $this->assertCount(2, $service->getXmlElement()->url);
+        $this->assertStringNotContainsString('robots.txt', $service->getXml());
+    }
+
+    public function testGenerateAddsNonHtmlPagesWithSitemapFrontMatterSetToTrue()
+    {
+        Routes::addRoute(new Route(new InMemoryPage('robots.txt', ['sitemap' => true])));
+
+        $service = new SitemapGenerator();
+        $service->generate();
+
+        $this->assertCount(3, $service->getXmlElement()->url);
+        $this->assertStringContainsString('robots.txt', $service->getXml());
+    }
+
     public function testGetXmlReturnsXmlString()
     {
         $service = new SitemapGenerator();

packages/framework/tests/Feature/SitemapFeatureTest.php

@@ -50,6 +50,7 @@ public function testTheSitemapFeature()
     protected function setUpBroadSiteStructure(): void
     {
         $this->file('_pages/about.md', "# About\n\nThis is the about page.");
+        $this->file('_pages/secret.md', "---\nsitemap: false\n---\n\n# Secret\n\nThis page is excluded from the sitemap.");
         $this->file('_pages/contact.html', '<h1>Contact</h1><p>This is the contact page.</p>');
         $this->file('_posts/hello-world.md', "# Hello, World!\n\nThis is the first post.");
         $this->file('_posts/second-post.md', "# Second Post\n\nThis is the second post.");
@@ -123,12 +124,6 @@ protected function expected(string $version): string
                 <changefreq>daily</changefreq>
                 <priority>0.9</priority>
             </url>
-            <url>
-                <loc>https://example.com/docs/search.json</loc>
-                <lastmod>2024-01-01T12:00:00+00:00</lastmod>
-                <changefreq>weekly</changefreq>
-                <priority>0.5</priority>
-            </url>
             <url>
                 <loc>https://example.com/docs/search.html</loc>
                 <lastmod>2024-01-01T12:00:00+00:00</lastmod>

packages/framework/tests/Unit/Pages/BladePageUnitTest.php

@@ -86,6 +86,12 @@ public function testShowInNavigation()
         $this->assertTrue((new BladePage())->showInNavigation());
     }
 
+    public function testShowInSitemap()
+    {
+        $this->assertTrue((new BladePage())->showInSitemap());
+        $this->assertFalse((new BladePage('foo', ['sitemap' => false]))->showInSitemap());
+    }
+
     public function testNavigationMenuPriority()
     {
         $this->assertSame(999, (new BladePage())->navigationMenuPriority());

packages/framework/tests/Unit/Pages/DocumentationPageUnitTest.php

@@ -88,6 +88,12 @@ public function testShowInNavigation()
         $this->assertTrue((new DocumentationPage())->showInNavigation());
     }
 
+    public function testShowInSitemap()
+    {
+        $this->assertTrue((new DocumentationPage())->showInSitemap());
+        $this->assertFalse((new DocumentationPage('foo', ['sitemap' => false]))->showInSitemap());
+    }
+
     public function testNavigationMenuPriority()
     {
         $this->assertSame(999, (new DocumentationPage())->navigationMenuPriority());

packages/framework/tests/Unit/Pages/HtmlPageUnitTest.php

@@ -122,6 +122,12 @@ public function testShowInNavigation()
         $this->assertTrue((new HtmlPage())->showInNavigation());
     }
 
+    public function testShowInSitemap()
+    {
+        $this->assertTrue((new HtmlPage())->showInSitemap());
+        $this->assertFalse((new HtmlPage('foo', ['sitemap' => false]))->showInSitemap());
+    }
+
     public function testNavigationMenuPriority()
     {
         $this->assertSame(999, (new HtmlPage())->navigationMenuPriority());

packages/framework/tests/Unit/Pages/InMemoryPageUnitTest.php

@@ -126,6 +126,21 @@ public function testShowInNavigation()
         $this->assertTrue((new InMemoryPage('foo'))->showInNavigation());
     }
 
+    public function testShowInSitemap()
+    {
+        $this->assertTrue((new InMemoryPage('foo'))->showInSitemap());
+        $this->assertFalse((new InMemoryPage('foo', ['sitemap' => false]))->showInSitemap());
+    }
+
+    public function testShowInSitemapIsFalseForPagesWithNonHtmlOutputPaths()
+    {
+        $this->assertFalse((new InMemoryPage('robots.txt'))->showInSitemap());
+        $this->assertFalse((new InMemoryPage('data.json'))->showInSitemap());
+        $this->assertFalse((new InMemoryPage('custom.xml'))->showInSitemap());
+
+        $this->assertTrue((new InMemoryPage('robots.txt', ['sitemap' => true]))->showInSitemap());
+    }
+
     public function testNavigationMenuPriority()
     {
         $this->assertSame(999, (new InMemoryPage('foo'))->navigationMenuPriority());

packages/framework/tests/Unit/Pages/MarkdownPageUnitTest.php

@@ -124,6 +124,12 @@ public function testShowInNavigation()
         $this->assertTrue((new MarkdownPage())->showInNavigation());
     }
 
+    public function testShowInSitemap()
+    {
+        $this->assertTrue((new MarkdownPage())->showInSitemap());
+        $this->assertFalse((new MarkdownPage('foo', ['sitemap' => false]))->showInSitemap());
+    }
+
     public function testNavigationMenuPriority()
     {
         $this->assertSame(999, (new MarkdownPage())->navigationMenuPriority());

packages/framework/tests/Unit/Pages/MarkdownPostUnitTest.php

@@ -124,6 +124,12 @@ public function testShowInNavigation()
         $this->assertFalse((new MarkdownPost())->showInNavigation());
     }
 
+    public function testShowInSitemap()
+    {
+        $this->assertTrue((new MarkdownPost())->showInSitemap());
+        $this->assertFalse((new MarkdownPost('foo', ['sitemap' => false]))->showInSitemap());
+    }
+
     public function testNavigationMenuPriority()
     {
         $this->assertSame(10, (new MarkdownPost())->navigationMenuPriority());

packages/testing/src/Common/BaseHydePageUnitTest.php

@@ -92,6 +92,8 @@ abstract public function testGetRoute();
 
     abstract public function testShowInNavigation();
 
+    abstract public function testShowInSitemap();
+
     abstract public function testGetSourcePath();
 
     abstract public function testGetLink();