feat: experimentalStreaming mode (#634)

Author: harlan-zw

docs/content/2.advanced/2.performance.md

@@ -60,16 +60,48 @@ export default defineNuxtConfig({
 ```
 
 Additionally, you may want to consider the following experimental options that may help with performance:
-- `experimentalCompression` - Gzips and streams the sitemap
-- `experimentalWarmUp` - Creates the sitemaps when Nitro starts
+
+- `experimentalStreaming`: Streams XML serialization in roughly 64 KB chunks instead of building the complete XML string in memory
+- `experimentalCompression`: Streams gzip or deflate compression when the client supports it
+- `experimentalWarmUp`: Creates the sitemaps when Nitro starts
+
+To reduce peak XML serialization memory while retaining compressed responses, enable streaming and compression together:
+
+```ts [nuxt.config.ts]
+export default defineNuxtConfig({
+  sitemap: {
+    experimentalStreaming: true,
+    experimentalCompression: true,
+  },
+})
+```
+
+Streaming applies to dynamic XML serialization. URL source fetching, normalization, filtering, deduplication, sorting, and the `sitemap:resolved` hook finish before the first XML chunk. The resolved URL plan remains in memory. Prerendering and `zeroRuntime` produce static files instead.
+
+Keep URL sources in their documented JSON formats. Streaming a source response does not make URL resolution incremental because the global processing stages require the complete URL plan.
+
+When `cacheMaxAgeSeconds` is enabled in production, streaming caches this finalized URL plan with stale-while-revalidate. Source resolution and the `sitemap:resolved` hook run on cache misses and refreshes; XML serialization and optional compression remain pull-driven for each response.
+
+The `sitemap:output` hook remains backwards compatible. If a hook reads or replaces `ctx.sitemap`, that response is buffered before being streamed to the client. Hooks that do not access `ctx.sitemap` keep the streaming serializer path.
+
+A CDN, reverse proxy, or deployment adapter may buffer the response after Nuxt sends it. Verify behavior in the production path rather than relying on local timing alone.
+
+### Verify the render mode
+
+Enable `debug` temporarily and inspect `X-Sitemap-Render-Mode`:
+
+- `stream`: XML serialization stayed pull driven
+- `buffered-hook`: A `sitemap:output` hook accessed the XML string
+
+A streamed response omits `Content-Length`. `Content-Encoding: gzip` or `deflate` confirms transport compression, but does not prove that XML serialization streamed.
 
 **Very large sites (100k+ URLs).** For sites at this scale, two practices matter most:
 
 1. **Cache the source endpoint.** Use `defineCachedEventHandler` on any `/api/*` route fed into `sources`. Without this, every cache miss (and every fresh chunk) re-hits your backend.
 
 2. **Set generous chunk sizes.** Search engines accept up to 50,000 URLs per file. The default `defaultSitemapsChunkSize` of 1000 generates 50× more chunks than necessary; bumping to `5000`–`50000` directly reduces total work and cache entries.
 
-Within a single sitemap, all chunks share one resolved-URLs computation (sources are fetched, normalized, and sorted once per `cacheMaxAgeSeconds` window — not once per chunk). Splitting one large sitemap into per-shard sitemaps (e.g. one per locale or content type) is still useful when shards have different cache lifetimes or different sources.
+With `cacheMaxAgeSeconds` enabled in production, chunks of the same base sitemap share one resolved URL computation per cache window. If caching is disabled, each requested chunk processes the complete base URL set before slicing its own URLs. Splitting one large sitemap into per-shard sitemaps, such as one per locale or content type, is still useful when shards have different cache lifetimes or sources.
 
 ## Zero Runtime Mode
 

docs/content/3.api/0.config.md

@@ -306,7 +306,15 @@ Whether to warm up the sitemaps when Nitro starts. This can be useful for large
 
 - Default: `false`
 
-Whether to compress and stream the sitemaps when the request accepts it.
+Whether to stream gzip or deflate compression when the request accepts it. This streams the transport even when XML serialization is buffered. Combine it with `experimentalStreaming` to stream both serialization and compression.
+
+## `experimentalStreaming: boolean`
+
+- Default: `false`
+
+Whether to serialize dynamic sitemap and sitemap index responses as a pull-driven stream. URL sources and the finalized URL plan are still resolved in full before serialization begins. Streaming a source response does not make source processing incremental.
+
+Prerendered output remains buffered. Accessing `ctx.sitemap` from the `sitemap:output` hook buffers that response for backwards compatibility. Enable `debug` to expose the `X-Sitemap-Render-Mode` response header. See the [performance guide](/docs/sitemap/advanced/performance#verify-the-render-mode) for constraints and verification.
 
 ## `credits: boolean`
 

docs/content/4.nitro-api/nitro-hooks.md

@@ -43,6 +43,8 @@ export default defineNitroPlugin((nitroApp) => {
 
 Triggered once the final structure of the XML is generated, provides the URLs as objects.
 
+With `experimentalStreaming` and production caching enabled, the finalized URL plan is cached. This hook runs on a plan cache miss or stale-while-revalidate refresh rather than on every sitemap response.
+
 For new URLs it's recommended to use `sitemap:input` instead. Use this hook for modifying entries or removing them.
 
 ```ts [server/plugins/sitemap.ts]
@@ -119,6 +121,8 @@ export default defineNitroPlugin((nitroApp) => {
 Triggered before the sitemap is sent to the client.
 It provides the sitemap as an XML string.
 
+When `experimentalStreaming` is enabled, the XML string is created lazily. Reading or replacing `ctx.sitemap` buffers the complete XML response. A hook that only observes the event or sitemap name preserves streaming serialization. With `debug` enabled, check `X-Sitemap-Render-Mode` for `stream` or `buffered-hook`.
+
 ```ts [server/plugins/sitemap.ts]
 import { defineNitroPlugin } from 'nitropack/runtime'
 

src/module.ts

@@ -106,6 +106,7 @@ export default defineNuxtModule<ModuleOptions>({
     cacheMaxAgeSeconds: 60 * 10, // cache for 10 minutes
     minify: false,
     debug: false,
+    experimentalStreaming: false,
     defaultSitemapsChunkSize: 1000,
     autoLastmod: false,
     discoverImages: true,
@@ -440,13 +441,15 @@ export default defineNuxtModule<ModuleOptions>({
     }
 
     // skip experimental runtime plugins in zeroRuntime mode
-    if (config.zeroRuntime && (config.experimentalWarmUp || config.experimentalCompression))
-      logger.warn('`experimentalWarmUp` and `experimentalCompression` are ignored in zeroRuntime mode.')
+    if (config.zeroRuntime && (config.experimentalWarmUp || config.experimentalCompression || config.experimentalStreaming))
+      logger.warn('`experimentalWarmUp`, `experimentalCompression`, and `experimentalStreaming` are ignored in zeroRuntime mode.')
     if (!config.zeroRuntime) {
       if (config.experimentalWarmUp)
         addServerPlugin(resolve('./runtime/server/plugins/warm-up'))
       if (config.experimentalCompression)
         addServerPlugin(resolve('./runtime/server/plugins/compression'))
+      if (config.experimentalStreaming || config.experimentalCompression)
+        addServerPlugin(resolve('./runtime/server/plugins/stream-transport'))
     }
 
     // @ts-expect-error untyped
@@ -782,6 +785,7 @@ export default defineNuxtModule<ModuleOptions>({
       isMultiSitemap: usingMultiSitemaps,
       excludeAppSources: config.excludeAppSources,
       cacheMaxAgeSeconds: nuxt.options.dev ? 0 : config.cacheMaxAgeSeconds,
+      experimentalStreaming: config.experimentalStreaming,
 
       autoLastmod: config.autoLastmod,
       defaultSitemapsChunkSize: config.defaultSitemapsChunkSize,

src/runtime/server/plugins/compression.ts

@@ -1,27 +1,89 @@
 import type { H3Event } from 'h3'
-import { getRequestHeader, setResponseHeader } from 'h3'
+import { appendResponseHeader, getRequestHeader, getResponseHeader, removeResponseHeader, setResponseHeader } from 'h3'
 import { defineNitroPlugin } from 'nitropack/runtime'
+import { logger } from '../../utils-pure'
+import { hasNonIdentityEncoding, isReadableStream, negotiateCompressionEncoding } from '../sitemap/stream'
 
-function getPreferredEncoding(event: H3Event): 'gzip' | 'deflate' | null {
-  const acceptEncoding = getRequestHeader(event, 'accept-encoding') || ''
-  if (acceptEncoding.includes('gzip'))
-    return 'gzip'
-  if (acceptEncoding.includes('deflate'))
-    return 'deflate'
-  return null
+let warnedAboutCompressionStream = false
+const NODE_COMPRESSION_INPUT_BATCH_BYTES = 512 * 1024
+
+function toByteStream(body: unknown): ReadableStream<Uint8Array> {
+  if (isReadableStream(body))
+    return body
+  if (body instanceof Blob)
+    return body.stream()
+
+  const value = typeof body === 'string' || body instanceof ArrayBuffer || ArrayBuffer.isView(body)
+    ? body
+    : JSON.stringify(body)
+  return new Blob([value as BlobPart]).stream()
+}
+
+function createNodeCompressionInputStream(source: ReadableStream<Uint8Array>): ReadableStream<Uint8Array> {
+  const reader = source.getReader()
+  let bytesRead = 0
+
+  return new ReadableStream<Uint8Array>({
+    async pull(controller) {
+      if (bytesRead >= NODE_COMPRESSION_INPUT_BATCH_BYTES) {
+        bytesRead = 0
+        await new Promise<void>(resolve => setTimeout(resolve, 0))
+      }
+
+      const result = await reader.read()
+      if (result.done) {
+        controller.close()
+        return
+      }
+
+      bytesRead += result.value.byteLength
+      controller.enqueue(result.value)
+    },
+    cancel(reason) {
+      return reader.cancel(reason)
+    },
+  })
+}
+
+function addVaryAcceptEncoding(event: H3Event) {
+  const vary = getResponseHeader(event, 'Vary')
+  const values = (Array.isArray(vary) ? vary : [vary])
+    .flatMap(value => String(value || '').split(','))
+    .map(value => value.trim().toLowerCase())
+  if (!values.includes('*') && !values.includes('accept-encoding'))
+    appendResponseHeader(event, 'Vary', 'Accept-Encoding')
 }
 
 export default defineNitroPlugin((nitro) => {
   nitro.hooks.hook('beforeResponse', (event, response) => {
     if (!event.context._isSitemap || !response.body)
       return
 
-    const encoding = getPreferredEncoding(event)
+    addVaryAcceptEncoding(event)
+    if (hasNonIdentityEncoding(getResponseHeader(event, 'Content-Encoding')))
+      return
+
+    const encoding = negotiateCompressionEncoding(getRequestHeader(event, 'accept-encoding') || '')
     if (!encoding)
       return
 
-    const body = typeof response.body === 'string' ? response.body : JSON.stringify(response.body)
-    response.body = new Blob([body]).stream().pipeThrough(new CompressionStream(encoding))
+    if (typeof CompressionStream === 'undefined') {
+      if (!warnedAboutCompressionStream) {
+        warnedAboutCompressionStream = true
+        logger.warn('Sitemap compression was requested, but CompressionStream is unavailable in this runtime. Sending the uncompressed response.')
+      }
+      return
+    }
+
+    const compression = new CompressionStream(encoding) as unknown as TransformStream<Uint8Array, Uint8Array>
+    const source = toByteStream(response.body)
+    // Node's CompressionStream can drain synchronous input before exposing output.
+    // Yield in bounded batches so socket backpressure and cancellation reach the source.
+    const compressionInput = event.node.res?.socket
+      ? createNodeCompressionInputStream(source)
+      : source
+    response.body = compressionInput.pipeThrough(compression)
+    removeResponseHeader(event, 'Content-Length')
     setResponseHeader(event, 'Content-Encoding', encoding)
   })
 })

src/runtime/server/plugins/stream-transport.ts

@@ -0,0 +1,24 @@
+import { removeResponseHeader } from 'h3'
+import { defineNitroPlugin } from 'nitropack/runtime'
+import { logger } from '../../utils-pure'
+import { createNodeResponseStream, isReadableStream } from '../sitemap/stream'
+
+export default defineNitroPlugin((nitro) => {
+  nitro.hooks.hook('beforeResponse', (event, response) => {
+    const nodeResponse = event.node.res
+    if (!event.context._isSitemap
+      || !isReadableStream(response.body)
+      // Fetch/edge adapters use a socket-less response shim and can consume the
+      // Web stream directly. Only H3's live Node transport needs this adapter.
+      || !nodeResponse?.socket
+      || typeof nodeResponse?.write !== 'function'
+      || typeof nodeResponse?.once !== 'function') {
+      return
+    }
+
+    response.body = createNodeResponseStream(response.body, (error) => {
+      logger.error('Failed to cancel sitemap response stream after the client disconnected.', error)
+    })
+    removeResponseHeader(event, 'Content-Length')
+  })
+})

src/runtime/server/sitemap/builder/index-xml.ts

@@ -0,0 +1,83 @@
+import type { ModuleRuntimeConfig, NitroUrlResolvers, SitemapIndexEntry } from '../../../types'
+import { withQuery } from 'ufo'
+import { createChunkedXmlStream } from '../stream'
+import { escapeValueForXml } from './xml'
+
+export function* renderSitemapIndexXmlChunks(sitemaps: SitemapIndexEntry[], resolvers: NitroUrlResolvers, { version, xsl, credits, minify }: Pick<ModuleRuntimeConfig, 'version' | 'xsl' | 'credits' | 'minify'>, errorInfo?: { messages: string[], urls: string[] }): Generator<string> {
+  const NL = minify ? '' : '\n'
+  const I1 = minify ? '' : '    '
+  const I2 = minify ? '' : '        '
+
+  yield '<?xml version="1.0" encoding="UTF-8"?>'
+
+  if (xsl) {
+    let relativeBaseUrl = resolvers.relativeBaseUrlResolver?.(xsl) ?? xsl
+    if (errorInfo && errorInfo.messages.length > 0) {
+      relativeBaseUrl = withQuery(relativeBaseUrl, {
+        errors: 'true',
+        error_messages: errorInfo.messages,
+        error_urls: errorInfo.urls,
+      })
+    }
+    yield `${NL}<?xml-stylesheet type="text/xsl" href="${escapeValueForXml(relativeBaseUrl)}"?>`
+  }
+
+  yield `${NL}<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">${NL}`
+  for (let i = 0; i < sitemaps.length; i++) {
+    const entry = sitemaps[i]!
+    if (i > 0)
+      yield NL
+    yield `${I1}<sitemap>${NL}${I2}<loc>${escapeValueForXml(entry.sitemap)}</loc>${NL}`
+    if (entry.lastmod)
+      yield `${I2}<lastmod>${escapeValueForXml(entry.lastmod)}</lastmod>${NL}`
+    yield `${I1}</sitemap>`
+  }
+  yield `${NL}</sitemapindex>`
+
+  if (credits)
+    yield `${NL}<!-- XML Sitemap Index generated by @nuxtjs/sitemap v${version} at ${new Date().toISOString()} -->`
+}
+
+export function urlsToIndexXml(sitemaps: SitemapIndexEntry[], resolvers: NitroUrlResolvers, { version, xsl, credits, minify }: Pick<ModuleRuntimeConfig, 'version' | 'xsl' | 'credits' | 'minify'>, errorInfo?: { messages: string[], urls: string[] }) {
+  const NL = minify ? '' : '\n'
+  const I1 = minify ? '' : '    '
+  const I2 = minify ? '' : '        '
+  let sitemapXml = ''
+  for (const entry of sitemaps) {
+    if (sitemapXml)
+      sitemapXml += NL
+    sitemapXml += `${I1}<sitemap>${NL}${I2}<loc>${escapeValueForXml(entry.sitemap)}</loc>${NL}`
+    if (entry.lastmod)
+      sitemapXml += `${I2}<lastmod>${escapeValueForXml(entry.lastmod)}</lastmod>${NL}`
+    sitemapXml += `${I1}</sitemap>`
+  }
+
+  const xmlParts = ['<?xml version="1.0" encoding="UTF-8"?>']
+
+  if (xsl) {
+    let relativeBaseUrl = resolvers.relativeBaseUrlResolver?.(xsl) ?? xsl
+    if (errorInfo && errorInfo.messages.length > 0) {
+      relativeBaseUrl = withQuery(relativeBaseUrl, {
+        errors: 'true',
+        error_messages: errorInfo.messages,
+        error_urls: errorInfo.urls,
+      })
+    }
+    xmlParts.push(`<?xml-stylesheet type="text/xsl" href="${escapeValueForXml(relativeBaseUrl)}"?>`)
+  }
+
+  xmlParts.push(
+    '<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">',
+    sitemapXml,
+    '</sitemapindex>',
+  )
+
+  if (credits)
+    xmlParts.push(`<!-- XML Sitemap Index generated by @nuxtjs/sitemap v${version} at ${new Date().toISOString()} -->`)
+
+  return xmlParts.join(NL)
+}
+
+export function urlsToIndexXmlStream(sitemaps: SitemapIndexEntry[], resolvers: NitroUrlResolvers, config: Pick<ModuleRuntimeConfig, 'version' | 'xsl' | 'credits' | 'minify'>, errorInfo?: { messages: string[], urls: string[] }): ReadableStream<Uint8Array> {
+  return createChunkedXmlStream(renderSitemapIndexXmlChunks(sitemaps, resolvers, config, errorInfo))
+}