feat(indexnow): re-enable submission for the curated set

[harlan-zw]

❓ Type of change

• 📖 Documentation
• 🐞 Bug fix
• 👌 Enhancement
• ✨ New feature
• 🧹 Chore
• ⚠️ Breaking change

📚 Description

Experiment A (Bing as a control) needs IndexNow back. nuxt-ai-ready 2.x dropped its IndexNow support in #53, so nothing submits today. The old submitter sent 424k submissions for 143k pages and drew HTTP 429 for eight days.

This adds a small submitter that cannot flood by design:

flowchart LR
  T[indexnow-submit, hourly at :10] --> K{key file 200?}
  K -->|no| X[log reason, send nothing]
  K -->|yes| G{gate open?}
  G -->|closed| L[log reason, send nothing]
  G -->|open| S[curated sitemaps via SELF]
  S --> P[new or lastmod moved; run cap 200, day cap 1000]
  P --> W[write ledger row and pending strike]
  W --> I[POST api.indexnow.org]
  I -->|200 or 202| R[settle: record fingerprints, clear strikes]
  I -->|429, 5xx, thrown| B[settle: keep strike, store retry time]
  I -->|400 or 422| Q[halve next request, skip a lone bad URL]
  B -->|3 strikes| H[halt 24h or Retry-After, max 7 days]

Loading

• The URL set is the skills, pages, authors and sources sitemaps. No table scan, no tags.
• A URL goes out only if it is new or its sitemap lastmod moved. A bulk lastmod bump drains over days because of the caps.
• 429 stores a retry time (the larger of the schedule and Retry-After). Three failures halt the task for 24 hours. The reason goes to console.error, a wide event and sync_jobs.
• The ledger row and a pending strike are written before the POST, then settled. A thrown send or a D1 failure after an accepted batch still counts against the caps and adds a strike. Nothing is marked submitted until IndexNow accepts it.
• A 400 or 422 halves the next request until one URL stands alone. That URL is skipped until its lastmod moves. Other 4xx halt for 24 hours. A 200 character excerpt of the response body reaches the log and halt reason.
• Retry-After is honoured up to 7 days, on a backoff and on a halt.
• The key file is answered by a Nitro route (server/handlers/indexnow-key.ts, registered from INDEXNOW_KEY), so the Worker serves it whether or not the assets router runs. The static public/ copy is gone. The task and production smoke both require 200 with the key.
• Reuses the existing 10 * * * * trigger. Migration 0128 adds three small tables.

Not tried: a live submission to api.indexnow.org. After deploy, read indexnow_batches and the indexnow-submit row in sync_jobs once the first :10 tick has run.

🤖 AI disclosure: Harlan Agent Kit modified this description. My AI open-source policy.

[harlan-github-agent]

🤖 BLOCKED

Harlan Agent Kit posted this automated review. It is not Harlan's personal review or approval. AI open source policy. A person still decides the merge.

• Merge gate: BLOCKED. The pull request has merge conflicts.

• Review gate: Passed. No finding above 80/100.

• CI gate: Passed.

• Logged (25/100): A thrown run never updates sync_jobs, so /admin/integrity can show a stale ok status for a task that is failing hourly. View code Optional follow-up.

  • Ask an agent to verify and fix this finding

• Logged (20/100): Three consecutive isolated 400/422 URLs halt the whole submission task for 24 hours, a behaviour the described skip design does not mention. View code Optional follow-up.

  • Ask an agent to verify and fix this finding

Selected findings run after merge and open separate pull requests.

[harlan-github-agent]

Open (82/100): The new migration duplicates number 0128, which the base already uses, failing the repository's own migration-naming gate and the deploy that depends on it.

Proof: The base already carries migrations/0128_skill_trending_admissions.sql (present in 1a6e81f), so this head has two 0128_* files. Running the repo's own verifier on the directory returns {"_tag":"fail","issues":[{"_tag":"duplicate_number","nu.

Next: Rename 0128_indexnow.sql to the next free number, coordinating with the #322 claim recorded in RESERVED_MIGRATION_NUMBERS (scripts/lib/d1-schema-verifier.ts:251-254), and update that map.

Harlan Agent Kit posted this automated review. It is not Harlan's personal review or approval. AI open source policy.

[harlan-github-agent]

Harlan Agent Kit posted this automated review. AI open source policy.

Automated Review: BLOCKED.

Read the Review results