perf(registry): cut skilld-db rows read on repo skill counts

[harlan-zw]

🔗 Linked issue

Related to #182, #185, #186

❓ Type of change

• 👌 Enhancement

📚 Description

skilld-db reads went from 104M rows/day on 11 Aug to about 1B/day. That puts the account past the 25B/month D1 free read tier. Three statements did most of it between 11 and 13 Sep:

Statement	Rows read	Calls
Cluster membership window query	827M	1,330
listDuplicateCandidateRows	792M	1,347
Derived tag count json_each	177M	4,463

The shared cause is (SELECT COUNT(*) FROM skills WHERE owner/repo ...) in a select list, evaluated for every matching row before any LIMIT. That reads sum(c²) rows, and davila7/claude-code-templates alone has 875 Skills. The count is now read once per statement through a grouped join, or only for rows on the page. The duplicate candidate and sitemap caches go from 5 minutes to 1h plus a day of stale-while-revalidate. The derived tag counts come from one grouped scan per hour, because a 404 for an unknown slug was never cached and re-ran the scan on every request.

I first planned to read repos.repo_skill_count instead, but it is a different number. Sync stores the SKILL.md file count, and the paths that clear source_resolved never update it. On prod it disagrees with the live count for 7,159 of 8,618 repos, including 299 that do have resolved Skills. Swapping it in would change canonicalRepoSkillPath routing, so the exact count stays.

Rows read per call on prod skilld-db, before and after:

Statement	Before	After
Duplicate candidates, all	588,618	49,862
Duplicate candidates, supported	63,062	38,102
Sitemap, all	161,323	44,919
Typeahead	161,678	45,741
users/davila7/skills	781,291	28,708
Cluster index	222,046	25,898
Cluster detail testing, count + list	142,671	39,843
querySkills one per owner	279,600	61,938
Tag page testing	144,028	85,276

Loose ends:

• Handler cache lengths: the tag and cluster maxAge and swr changes are left to fix(clusters): add stale-while-revalidate window to cluster caches #189 and fix(registry): serve tag profiles stale-while-revalidate #190, which already touch those lines. The cost plan wanted 15 minutes or more on both. fix(clusters): add stale-while-revalidate window to cluster caches #189 currently has 60s fresh plus 5 min stale, which may be worth raising there.
• Merge conflicts: this PR will conflict textually with fix(registry): serve tag profiles stale-while-revalidate #190 and fix(api): cache user skills route reads per login #191 in the same handlers.
• Tie order: the tag page and querySkills now sort the page again by the same key. On the name and owner sorts, rows that tie on that key could come back in a different order.
• Not touched: community.ts also has the correlated count, but moving it saved nothing (344K to 343K), so its cost is elsewhere. The collection detail ranked_skills CTE (120M over 71 calls) is also out of scope.

🤖 AI disclosure: Harlan Agent Kit modified this description. My AI open-source policy.

https://claude.ai/code/session_014N85FDJLggfBs3BHJXT7GZ

[harlan-zw]

Checked by hand against prod skilld-db, since CI cannot cover it:

• Ran every old and new statement read-only on prod. Each pair returned the same rows in the same order, compared by a hash of the ordered rows: duplicate candidates (all and supported), sitemap, typeahead, users/davila7/skills, cluster index, cluster detail count and list for testing, querySkills stars and one-per-owner pages, and the tag page for testing.
• The hourly derived tag map matches the old per-slug COUNT(DISTINCT ...) for all 388 tags with 10 or more Skills, with 0 mismatches and no non-text tag values.
• repos.repo_skill_count differs from the live resolved count for 7,159 of 8,618 repos, which is why it is not used.
• A first draft bound the pinned cluster keys in the wrong order and returned 0 rows on prod. The new SQLite case in cluster-page-sql.test.ts fails on that order and passes on the fix.
• Not deployed. The daily rows-read drop still needs a D1 analytics check after release.

[harlan-github-agent]

🤖 MERGED

Harlan Agent Kit posted this automated review. It is not Harlan's personal review or approval. AI open source policy. Last updated: 2026-09-14 08:14 UTC.

GitHub merged this pull request.

No material findings were recorded.

The pull request closed.