feat(seo): index only Skills admitted from trending boards

[harlan-zw]

❓ Type of change

• 👌 Enhancement

📚 Description

Stacked on #317 (fix/soft-404-missing-skills). The base is that branch, not main.

SEO experiment, gate 2026-11-11. Google holds skilld's curated URLs as "Discovered, currently not indexed" and crawls about 29 HTML pages a day. So only Skills that appeared on a trending board (week, month, all) stay indexable and in the skills sitemap. Every other Skill page is noindex,follow and leaves the sitemap.

• One rule, one file. layers/registry/server/utils/trending-admission.ts names the experiment, the gate, and the cull path.
• Additive. Migration 0129_skill_trending_admissions.sql stores the admitted set with admitted_at. An hourly task, admit-trending-skills, adds Skills and never removes one, so a page cannot flip between index and noindex.
• Sitemap equals the indexable set. The Skill API and the sitemap share isSkillIndexable and SKILL_ADMITTED_SQL. The ?supported= variant is gone, because it made the sitemap narrower than the pages.
• Discovery. Each trending board lists admitted Skills that left it as plain links, with real ?page= links. The boards and /skills join the skills sitemap when they render index. The homepage links Week, Month, and All time.
• One robots decision. A Nitro plugin sets X-Robots-Tag from the robots meta the page rendered. The robots module set index, follow before render, on pages whose HTML said noindex,follow.

flowchart LR
  B[trending board loader<br/>week, month, fallback] --> J[admit-trending-skills<br/>hourly]
  L[leaderboard page 1<br/>all] --> J
  J --> T[(skill_trending_admissions)]
  T --> A[Skill API<br/>isSkillIndexable]
  T --> S[skills sitemap]
  T --> E[trending pages<br/>Earlier on this board]
  A --> M[robots meta]
  M --> H[X-Robots-Tag]

Loading

Size of the set (production D1, read only, 2026-09-30). Boards hold 95 unique Skills today (week 30, month 30, all 50). 65 of them pass the existing quality score and are not aggregators. That is the starting set, against 1,414 Skill URLs in the sitemap now. The set grows as boards move.

High-demand Skills against the rule (none added by hand):

Skill	In the rule	Why
humanizer (blader)	in	on a board, score passes
taste-skill (leonxlnx)	in	on a board, score passes
visual-explainer (nicobailon)	in	on a board, score passes
ui-ux-pro-max	out	not on a board, and seo_indexable = 0
caveman	out	not on a board, and seo_indexable = 0 in both repos
ponytail (dietrichgebert)	out	passes the score, not on a board
stop-slop (hardikpandya)	out	passes the score, not on a board

Probe exceptions (experiment D, same gate, 2026-11-11). Two named pages stay indexable whatever their quality score, so harlanzw.com posts can link to them. They are rows with first_board = 'probe', listed in PROBE_EXCEPTIONS. Cull path: empty that list and DELETE FROM skill_trending_admissions WHERE first_board = 'probe'.

• /gh/harlan-zw/gscdump is a single-Skill repository, so its hub URL is the Skill page. The registry holds it with seo_indexable = 0. The probe row admits it.
• /gh/harlan-zw/nuxt-seo/nuxtjs-seo replaces nuxtseo-cli, which lives in a private repository that skilld can never list. nuxtjs-seo is public and in the production registry (the Skill API answers 200; its page is noindex today). The 0129 seed admits it as a probe row.

Hygiene.

• Tag chips link to the final page, not to a 301. Related tags skip redirected tags.
• Related Skills and tag lists skip Skills whose SKILL.md is gone, so pages stop linking to 410s.
• A repository hub is indexable only when it renders its source. A renamed repo (/gh/hyf0/vue-skills) shows "Source not found", so it is noindex and leaves the sources sitemap once the repo row stores the new GitHub identity. No 301: the registry keys Skills by the old name, so the new name has no page.
• /skills/trending?page=N has its own canonical only for 1 < N <= the admitted list's page count. Page 1 canonicalises to the bare range. A page past the last answers 404, so old /skills/leaderboard?page=2..5 links stop resolving.
• The pages sitemap skips any /agents/* page the admission rule refuses, found from the pages folder.
• The Skill page and the sitemap share one duplicate decision, aggregator copies included. Admitted-Skill links on the trending pages come from the sitemap set, so they never point at a noindex page.
• /make-skill leaves the pages sitemap. The 8 tag pages go noindex and the tags sitemap is removed. That last one is a judgement call; revert [slug].get.ts and __sitemap__/tags.ts to undo it.

Freeze audit (VISION principle 2). layers/marketing/app/utils/page-admissions.ts records target query and bar. An audited page with no entry is noindex,follow and leaves the sitemap. Every /agents/* page is audited by default. The bar is 100 measured searches a month; I chose it, so adjust it if you disagree.

Page	Target query	Volume, KD	Result
/agents/codex	codex skills	4,400, 4	index
/agents/cursor	cursor skills	2,900, 7	index
/agents/claude-code	claude code agent skill	390	index
/vs/context7	context7 vs skills	none	index, template for experiment C
/agents	none	none	noindex
/agents/gemini-cli, github-copilot, hermes, openclaw, opencode, windsurf	none above 40	under 100	noindex
/vs/skills-sh	skill sh (navigational)	3,600	noindex
/verify, /docs/cli, /privacy	none	none	noindex
/learn/private-repositories, /learn/author-* (6)	none	none	noindex

Cull path for an admitted page: remove its entry.

📝 Migration

1. Apply 0129 through the normal migration flow. It must merge after feat(indexnow): re-enable submission for the curated set #320 (IndexNow, 0128_indexnow.sql). Renumber it if feat(indexnow): re-enable submission for the curated set #320 merges later. Until feat(indexnow): re-enable submission for the curated set #320 lands, the migration-naming test reports a gap at 0128, so CI fails on that one test.
2. The migration seeds today's 65 admitted Skills (a production snapshot) and the two probe rows, so the set starts populated. While the table holds no row at all, the rule falls back to the pre-experiment seo_indexable decision and logs trending-admission / fallback-empty-set once per isolate. In that state the sitemap follows the page rule, so it is wider than the old supported-only sitemap.
3. To end the experiment: DELETE FROM skill_trending_admissions empties the set, and reverting isSkillIndexable restores the old rule.

🤖 AI disclosure: Harlan Agent Kit modified this description. My AI open-source policy.

[harlan-zw]

Checked by hand, since CI cannot cover it. Local dev server, seeded D1, Googlebot user agent:

• Admitted Skill page and /agents/codex: x-robots-tag: index,follow and meta index,follow.
• Non-admitted Skill, /agents/windsurf, /make-skill: noindex,follow in both header and meta.
• Skills sitemap listed only admitted Skills, /skills, and the trending boards.
• Trending SSR: month board 12 board links plus 3 "Earlier on this board" links (15). Production today, before this change: 30, 30, 50 links for week, month, all, and no earlier list.
• Not run: hourly task in a deployed Worker, e2e suite.

[harlan-zw]

🤖 PENDING

Harlan Agent Kit posted this automated review. It is not Harlan's personal review or approval. AI open source policy. Human merge decision still required. Waiting on #320: the migration-naming test reports a gap at 0128 until 0128_indexnow.sql merges.

• Logged (65/100): /gh/hyf0/vue-skills and /gh/brianlovin/claude-config render "Source not found" to Googlebot, because GitHub reports a new identity and repoSource is null (gh/[owner]/[repo]/index.vue:367). The robots line now reads fetchedRepoSource.seoIndexable (:306), so both pages turn index,follow in meta and header. Both are in the sources sitemap.
• Logged (55/100): ?page=N gets a self canonical with no upper bound (skills/trending.vue:87, :330). /skills/leaderboard?page=2 to ?page=5 redirect to /skills/trending?range=all&page=N, which now render index,follow copies of ?range=all with their own canonical. nuxt.config.ts:547 records that the canonical used to drop page.
• Logged (35/100): frozenNoindexPaths() returns only the listed paths (page-admissions.ts:90). A new /agents/* page renders noindex,follow through the prefix rule (:79) but stays in the pages sitemap.
• Logged (30/100): The Skill page builds duplicate groups with aggregators (skills-registry.ts:800); the sitemap builds them without (:853). If an aggregator copy is admitted and ranks higher, the page turns noindex with a canonical to a noindex page, and the sitemap still lists it.
• Logged (20/100): Comments cite notes/skilld-seo-death-zone-2026-09-30.md (trending-admission.ts:14) and ~/scratch/seo-deathzone-2026-09-30/gaps.csv (page-admissions.ts:12). Neither file is in the repository. AGENTS.md names this trap for GOOGLE_RECOVERY.md.
• Logged (15/100): The listAdmittedSkills comment says every link points at an indexable page (trending-admission.ts:270). The query skips the broken-repository and duplicate checks that the page applies.
• Logged (10/100): The new constants separate the DEFAULT_TRENDING_RANGE doc comment from its declaration (shared/trending-range.ts:19). The tag page comment still points at the deleted __sitemap__/tags.ts (tags/[slug].get.ts:217).

[harlan-github-agent]

🤖 MERGED

Harlan Agent Kit posted this automated review. It is not Harlan's personal review or approval. AI open source policy. Last updated: 2026-09-30 09:50 UTC.

GitHub merged this pull request.

• Logged (25/100): A failed admitted-list fetch makes valid paginated board URLs answer 404. View code
• Logged (20/100): The robots meta detector matches any meta whose name merely starts with 'robots'. View code

0b599464-d768-4eea-9a2c-342264c0f2a9