Bugs and doc mismatches found while testing 6.2.4 #338

🤖 An agent (Claude Code) filed this issue while writing the @nuxtjs/robots package Skill with skilld's generate-package-skill. It reproduced each item against @nuxtjs/robots 6.2.4.

Setup for every repro: @nuxtjs/robots 6.2.4 from npm, nuxt 4.6.0, Node 24.18.0, pnpm. Each fixture is a fresh consumer app with modules: ['@nuxtjs/robots'] and site: { url: 'https://example.com' }. "Server" means nuxt build and then node .output/server/index.mjs.

"Main" says if current main (with #333 and #335) still has the same code. I read main. I did not run it.

The items are ordered by impact. Silent wrong results come first, then doc mismatches, then cleanups. One item that main already fixes is at the end.

Silent wrong results

1. definePageMeta({ robots: false }) has no effect

Expected (per docs): the page gets noindex, nofollow in the header and the meta tag.

Observed: the page stays indexable in production, and in dev with ?mockProductionEnv. A dynamic page is stored as /users/:id(), which never equals a request path.

<!-- app/pages/hidden.vue -->
<script setup lang="ts">
definePageMeta({ robots: false })
</script>

<template>
  <div>hidden</div>
</template>

curl -sI localhost:3000/hidden | grep -i x-robots-tag
# x-robots-tag: index, follow, max-image-preview:large, max-snippet:-1, max-video-preview:-1

Logging hooks in nuxt.config shows the order. nitro:config sees pageMetaRobots: {}. Then pages:resolved sees /hidden false and /users/:id() false. The map is filled after Nitro copied the runtime config.

• Source: src/module.ts#L271, src/module.ts#L546, getPathRobotConfig.ts#L133
• Docs: Disable Page Indexing, Page Meta
• Main: same code.
• Suggestion: turn page meta into route rules at pages:resolved, so dynamic paths match too.

2. The per-host robots:config recipe leaks to other hosts

Expected (per docs): the hook disallows / for staging hosts only.

Observed: after one staging request to /robots.txt, the production host serves noindex, nofollow on every page. This lasts until another host fetches /robots.txt. Staging pages are not noindexed unless a staging robots.txt request came first. The staging robots.txt also says (indexable) above Disallow: /.

// server/plugins/robots-domain.ts (copied from the docs)
export default defineNitroPlugin((nitroApp) => {
  nitroApp.hooks.hook('robots:config', (ctx) => {
    const host = ctx.event?.headers.get('host')
    if (host?.includes('staging') || host?.includes('test')) {
      ctx.groups[0].disallow = ['/']
    }
  })
})

curl -sI -H 'Host: example.com' localhost:3000/ | grep -i x-robots-tag
# x-robots-tag: index, follow, max-image-preview:large, max-snippet:-1, max-video-preview:-1
curl -s -H 'Host: staging.example.com' localhost:3000/robots.txt
# # START nuxt-robots (indexable)
# User-agent: *
# Disallow: /
# # END nuxt-robots
curl -sI -H 'Host: example.com' localhost:3000/ | grep -i x-robots-tag
# x-robots-tag: noindex, nofollow

A site-config:init hook that pushes { indexable: false } for the staging host works per request, with no leak.

• Source: src/runtime/server/util.ts#L21 stores the per-request result on nitroApp.
• Docs: Disabling Site Indexing, Handling Multiple Domains, Nitro Hooks
• Main: same code, same docs.
• Suggestion: keep the request context on the event, or point the docs at site-config:init.

3. NUXT_SITE_ENV=staging at nuxt generate writes no site-wide noindex header

Expected (per docs): set NUXT_SITE_ENV or NUXT_SITE_INDEXABLE at build time, and static files get the noindex header.

Observed: robots.txt and the meta tags are noindex. _headers keeps the per-route rules of an indexable site and has no /* rule. NUXT_SITE_INDEXABLE=false works.

The fixture has the route rules '/secret/**': { robots: false } and '/ai': { robots: { index: true, noai: true } }.

NUXT_SITE_ENV=staging NITRO_PRESET=netlify-static nuxt generate
cat dist/_headers
# /_nuxt/builds/meta/*
#   cache-control: public, max-age=31536000, immutable
# /_nuxt/builds/*
#   cache-control: public, max-age=1, immutable
# /secret/*
#   X-Robots-Tag: noindex, nofollow
# /ai
#   X-Robots-Tag: index, noai
# /_nuxt
#   X-Robots-Tag: noindex
# /_nuxt/*
#   cache-control: public, max-age=31536000, immutable
#   X-Robots-Tag: noindex

NUXT_SITE_INDEXABLE=false NITRO_PRESET=netlify-static nuxt generate
grep -B1 X-Robots-Tag dist/_headers
# /*
#   X-Robots-Tag: noindex, nofollow

• Source: src/module.ts#L470 reads useSiteConfig().indexable !== false, which ignores env.
• Docs: How Nuxt Robots Works
• Main: same code.
• Suggestion: derive build-time indexability from env too.

4. Content frontmatter robots sets the meta tag but not X-Robots-Tag

Expected (per docs): frontmatter configures robots for the page. Route rules and useRobotsRule() set both the header and the meta tag.

Observed: the meta tag says noindex, nofollow, and the header says index. @nuxt/content 3.16.1, zod 4.6.5.

<!-- content/hidden.md -->
---
robots: false
---

The page is the docs' catch-all page with useSeoMeta(page.value?.seo || {}).

curl -s -D - localhost:3000/hidden | grep -i 'x-robots-tag\|name="robots"'
# x-robots-tag: index, follow, max-image-preview:large, max-snippet:-1, max-video-preview:-1
# <meta name="robots" content="noindex, nofollow">

• Source: src/module.ts#L414 sets only seo.robots.
• Docs: Nuxt Content
• Main: same code.
• Suggestion: send the header too, or document useRobotsRule(page.value?.robots ?? undefined).

5. disableNuxtContentIntegration is ignored

Expected (per docs and types): robots: { disableNuxtContentIntegration: true } turns off the frontmatter mapping.

Observed: /hidden from item 4 still renders <meta name="robots" content="noindex, nofollow">.

• Source: src/module.ts#L154 declares the option. Nothing reads it.
• Docs: Nuxt Content, Config
• Main: same code.
• Suggestion: check the option before registering the content:file:afterParse hook.

6. Without a direct zod 4, frontmatter robots: false becomes the string "false"

Expected: robots: false gives noindex, or the build warns.

Observed: in a clean install without zod, defineRobotsSchema() resolves the zod 3.25.76 that @nuxt/content 3.16.1 depends on. The page then sends false as the rule. Nothing warns. After pnpm add zod (4.6.5), the same page sends noindex, nofollow.

pnpm add nuxt@4.6.0 @nuxtjs/robots@6.2.4 @nuxt/content@3.16.1

// content.config.ts (from the docs)
import { defineCollection, defineContentConfig } from '@nuxt/content'
import { defineRobotsSchema } from '@nuxtjs/robots/content'
import { z } from 'zod'

export default defineContentConfig({
  collections: {
    content: defineCollection({
      type: 'page',
      source: '**/*.md',
      schema: z.object({ robots: defineRobotsSchema() }),
    }),
  },
})

The catch-all page calls useSeoMeta(page.value?.seo || {}) and useRobotsRule(page.value?.robots ?? undefined).

curl -s -D - localhost:3000/hidden | grep -i 'x-robots-tag\|name="robots"'
# x-robots-tag: false
# <meta name="robots" content="false">

• Source: src/content.ts#L9. The zod peer is >=3 and optional.
• Docs: Nuxt Content
• Main: the peer is now ^4.6.5 and still optional. The schema code is the same. Not run on main.
• Suggestion: warn when the resolved zod is v3, or map "true" and "false" to booleans.

7. useRobotsRule(null) sends X-Robots-Tag: null

Expected (per types): null is not a RobotsValue. Content returns null for an unset frontmatter key, so it reaches the composable at runtime. It should keep the default rule, like undefined.

Observed: the header is the literal string null, and the meta tag is removed.

<!-- app/pages/[...slug].vue, for a Markdown file with no robots key -->
<script setup lang="ts">
const route = useRoute()
const { data: page } = await useAsyncData(`page-${route.path}`, () => queryCollection('content').path(route.path).first())
useSeoMeta(page.value?.seo || {})
useRobotsRule(page.value?.robots)
</script>

curl -s -D - localhost:3000/open | grep -i 'x-robots-tag\|name="robots"'
# x-robots-tag: null

• Source: useRobotsRule.ts#L41 skips only undefined. Line 58 sends null as a string.
• Main: same code.
• Suggestion: treat null like undefined.

8. app/assets/robots.txt and app/pages/robots.txt are ignored

Expected (per docs): assets/robots.txt and pages/robots.txt are merged. In the Nuxt 4 layout, those folders live in app/.

Observed: files in app/assets/ and app/pages/ are not merged, and nothing is logged. The same file in a root assets/ folder is merged.

printf 'User-agent: *\nDisallow: /from-app-assets\n' > app/assets/robots.txt
nuxt build && curl -s localhost:3000/robots.txt
# # START nuxt-robots (indexable)
# User-agent: *
# Disallow:
# # END nuxt-robots

• Source: src/module.ts#L315 resolves from rootDir, not srcDir.
• Docs: Config using Robots.txt
• Main: same code.
• Suggestion: resolve assets and pages from srcDir.

Doc mismatches

9. A string comment in a group prints one line per character

Expected (per docs and types): comment?: Arrayable<string>. The docs use a string.

Observed:

robots: { groups: [{ userAgent: ['Googlebot'], disallow: ['/private'], comment: 'Google only' }] }

curl -s localhost:3000/robots.txt
# # G
# # o
# # o
# # g
# # l
# # e
# ...
# User-agent: Googlebot
# Disallow: /private

• Source: src/util.ts#L382 does not wrap comment in asArray. Line 419 iterates the string.
• Docs: Config using Nuxt Config, Config
• Main: same code, same docs.
• Suggestion: normalize comment with asArray.

10. disallowNonIndexableRoutes disallows /_nuxt and noai routes

Expected (per docs): route rules that disallow indexing are added to robots.txt.

Observed: the module's own build-asset rule is added too. Then the build warns about it. A route with { index: true, noai: true } is also disallowed, though the docs present it as an indexed page.

robots: { disallowNonIndexableRoutes: true },
routeRules: {
  '/secret/**': { robots: false },
  '/ai': { robots: { index: true, noai: true } },
},

nuxt build
# [@nuxt/robots] WARN You have disallowed robots accessing /_nuxt/**, this may prevent your site from being indexed correctly.
curl -s localhost:3000/robots.txt
# User-agent: *
# Disallow:
# Disallow: /secret/*
# Disallow: /ai
# Disallow: /_nuxt
# Disallow: /_nuxt/*

getPathRobotConfig(event, { path: '/ai' }) also returns indexable: false.

• Source: src/module.ts#L464, src/module.ts#L485, src/util.ts#L704
• Docs: Config Using Route Rules, Config
• Main: same code.
• Suggestion: skip the module's own /_nuxt rules, and stop counting noai and noimageai as noindex.

11. The docs spell the option blockAIBots

Expected: the key is blockAiBots.

Observed: the recipes page prose says blockAIBots. The code block on the same page is correct.

• Docs: Robots.txt Recipes, and docs/content/6.releases/4.v5.md.
• Main: same docs.
• Suggestion: fix the prose.

12. The robotsTxt JSDoc says @default false

Expected: the JSDoc matches the default.

Observed: the JSDoc says @default false. The module defaults set robotsTxt: true, and robots.txt is served by default.

• Source: src/module.ts#L155-L160
• Main: same code.
• Suggestion: change it to @default true.

13. The Nitro API docs import from #imports, which fails nuxt typecheck on Nuxt 4.6

Expected: the docs examples typecheck.

Observed: with Nuxt 4.6.0, vue-tsc 3.3.12, and TypeScript 6.0.3, a server route and a server plugin that import from #imports fail. The same files with no import line, using Nitro auto-imports, pass.

nuxt typecheck
# server/api/robots-check.get.ts(2,10): error TS2305: Module '"#imports"' has no exported member 'defineEventHandler'.
# server/api/robots-check.get.ts(2,30): error TS2305: Module '"#imports"' has no exported member 'getPathRobotConfig'.
# server/plugins/robots-host.ts(4,35): error TS7006: Parameter 'nitroApp' implicitly has an 'any' type.

• Docs: getSiteRobotConfig(), getPathRobotConfig()
• Main: same docs.
• Suggestion: drop the #imports lines in the server examples.

14. getBotDetection(headers) reads only a lowercase user-agent key

Expected (per docs and types): the docs say "Works with any headers object". The type is Record<string, string | string[] | undefined>.

Observed:

import { getBotDetection } from '@nuxtjs/robots/util'

const ua = 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)'
getBotDetection({ 'User-Agent': ua }) // { isBot: false }
getBotDetection(new Headers({ 'user-agent': ua })) // { isBot: false }
getBotDetection({ 'user-agent': ua }) // { isBot: true, botName: 'googlebot', ... }

• Source: src/util.ts#L533
• Docs: getBotDetection()
• Main: same code.
• Suggestion: read the key without case, and accept a Headers instance.

15. useRobotsRule(true) overrides a site that is not indexable

Expected (per docs): "Providing a boolean will either enable or disable indexing for the current path using the default rules." The docs do not say that true beats site.indexable: false.

Observed: with NUXT_SITE_ENV=staging, every page sends noindex, nofollow, except a page that calls useRobotsRule(true).

<script setup lang="ts">
useRobotsRule(true)
</script>

NUXT_SITE_ENV=staging node .output/server/index.mjs
curl -sI localhost:3000/search | grep -i x-robots-tag
# x-robots-tag: index, follow, max-image-preview:large, max-snippet:-1, max-video-preview:-1

• Source: useRobotsRule.ts#L49
• Docs: useRobotsRule()
• Main: same code.
• Suggestion: document the override, or keep noindex when the site is not indexable.

16. With debug: true in production, the hints say indexing is blocked in development

Expected: hints match the environment.

Observed: robots: { debug: true }, production server:

curl -s localhost:3000/__robots__/debug.json
# {"robotsTxt":"# START nuxt-robots (indexable)\n...","indexable":true,"hints":["Indexing is blocked in development. You can mock a production environment with ?mockProductionEnv query."], ...}

• Source: getSiteRobotConfig.ts#L26
• Main: same code.
• Suggestion: add this hint only in dev.

Cleanups

17. The i18n docs leave out the unprefixed path

With strategy: 'prefix' and disallow: ['/admin'], robots.txt has Disallow: /admin, Disallow: /en/admin, and Disallow: /fr/admin. The docs show only the prefixed lines. @nuxtjs/i18n 10.6.0.

• Docs: Nuxt I18n
• Main: same docs.
• Suggestion: show the unprefixed line in the example output.

18. The Content load-order rule does not reproduce

The docs say robots "must" load before @nuxt/content. With @nuxt/content 3.16.1, both orders give the same meta tag. The load-order warning did not print in either order.

• Source: src/module.ts#L400
• Docs: Nuxt Content
• Main: same code, same docs.
• Suggestion: drop the rule if Content v3 no longer needs it, or fix the warning check.

19. The "valid paths" log lists ./public/_robots.txt twice

When the module moves public/robots.txt, the info log lists ./public/_robots.txt twice.

• Source: src/module.ts#L317 and #L319
• Main: same code.
• Suggestion: remove the duplicate entry.

Fixed on main, not yet released: a build that calls useBotDetection() fails with Rolldown failed to resolve import "@vueuse/core" from .../dist/runtime/app/utils/fingerprinting.js, because 6.2.4 lists @vueuse/core only in devDependencies (package.json#L102).