Skip to content

perf(collections): rank only the collection's repositories - #215

Merged
harlan-zw merged 1 commit into
mainfrom
perf-skilld-rows-read-2
Sep 14, 2026
Merged

harlan-zw merged 1 commit into
mainfrom
perf-skilld-rows-read-2

Conversation

@harlan-zw

Copy link
Copy Markdown
Collaborator

Why

Collection detail was the largest skilld-db statement that #212 left alone. It ranked every Skill in the registry and ran a correlated repo Skill count per row, then joined the collection.

Statement 7d rows read (7 to 14 Sep) Calls Rows per call
collection detail ranked_skills 291.6M 173 1,685,882

The manifest and index handlers use the same CTE without the count, at about 64K rows per call and low volume.

Change

  • The CTE now keeps only the collection's repositories. rn partitions by repository, so every rank stays the same.
  • Detail adds the repo Skill count to the joined rows, not every Skill.
  • Detail binds ?1 in both places. The index binds its id list twice.
  • No index, so no new write cost.

Verification

Production skilld-db, read-only. Result rows hashed in order.

Query Before After Rows
detail, collection 150 (largest, includes a repo entry) 1,692,948 637 8, same hash
manifest, collection 150 64,156 471 8, same hash
index, 7 collections 64,715 2,238 40, same hash

Projected saving: about 41M rows read a day.

  • New test/unit/collection-detail.test.ts covers a named Skill, a repository entry picking its newest Skill, and a repository outside the collection. Narrowing the new filter makes it fail.
  • pnpm typecheck: pass
  • eslint on changed files: pass
  • vitest run test/unit: 242 files, 1,690 tests pass

https://claude.ai/code/session_014N85FDJLggfBs3BHJXT7GZ

Collection detail ranked every Skill in the registry and ran a correlated
repo Skill count per row before joining the collection: 1.69M rows read per
call, 291.6M over 173 calls between 7 and 14 Sep 2026. The manifest and
index handlers ranked the whole table too, at about 64K rows per call.

The rank partitions by repository, so limiting the CTE to the collection's
repositories leaves every rank unchanged. On production data collection 150
returns identical rows at 637 rows read, manifest at 471, index at 2,238.

Claude-Session: https://claude.ai/code/session_014N85FDJLggfBs3BHJXT7GZ
@harlan-github-agent harlan-github-agent Bot added harlan-agent-running An Agent holds a Task on this issue or pull request right now. harlan-agent-review-required Pull request triage requires an adversarial Review for this head commit. labels Sep 14, 2026
@harlan-github-agent

harlan-github-agent Bot commented Sep 14, 2026 •

Copy link
Copy Markdown
Contributor

🤖 MERGED

Harlan Agent Kit posted this automated review. It is not Harlan's personal review or approval. AI open source policy. Last updated: 2026-09-14 14:04 UTC.

GitHub merged this pull request.

No material findings were recorded.

134b8ab9-1ccb-4249-b074-2ddaa40db840

@harlan-github-agent harlan-github-agent Bot added harlan-agent-pending The automated Review is waiting on a gate for this head commit. harlan-agent-ready The automated Review passed every gate on this head commit. and removed harlan-agent-running An Agent holds a Task on this issue or pull request right now. harlan-agent-review-required Pull request triage requires an adversarial Review for this head commit. harlan-agent-pending The automated Review is waiting on a gate for this head commit. harlan-agent-ready The automated Review passed every gate on this head commit. labels Sep 14, 2026
@harlan-zw
harlan-zw merged commit 10a6aa2 into main Sep 14, 2026
5 checks passed
@harlan-github-agent harlan-github-agent Bot removed the harlan-agent-ready The automated Review passed every gate on this head commit. label Sep 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant