Skip to content

perf(d1): bound scoring and discovery query work - #620

Merged
harlan-zw merged 1 commit into
mainfrom
fix/d1-slow-query-sweep
Oct 11, 2026
Merged

harlan-zw merged 1 commit into
mainfrom
fix/d1-slow-query-sweep

Conversation

@harlan-zw

Copy link
Copy Markdown
Collaborator

🔗 Linked issue

❓ Type of change

  • 📖 Documentation
  • 🐞 Bug fix
  • 👌 Enhancement
  • ✨ New feature
  • 🧹 Chore
  • ⚠️ Breaking change

📚 Description

Score recomputation recounts sibling Skills for each Skill. The live query recorded 335 million D1 rows read in 24 hours. Group live counts once. Load discovery metadata after pagination. Build tag candidates before joining Skills. Trim source text only for missing embeddings. Dirty drains also avoid a second scoring read.

A live comparison reduced full scoring reads from 13,951,401 to 165,855, with identical rows. No indexes or migrations are added.

D1 query changes: grouped scoring, bounded discovery reads, and deferred source trimming use existing tables and indexes

🤖 AI disclosure: Harlan Agent Kit modified this description. My AI open-source policy.

@harlan-github-agent harlan-github-agent Bot added harlan-agent-running An Agent holds a Task on this issue or pull request right now. harlan-agent-review-required Pull request triage requires an adversarial Review for this head commit. labels Oct 11, 2026
@harlan-github-agent

harlan-github-agent Bot commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

🤖 MERGED

Harlan Agent Kit posted this automated review. It is not Harlan's personal review or approval. AI open source policy. Last updated: 2026-10-11 01:39 UTC.

GitHub merged this pull request.

  • Logged (10/100): The PR's explanatory evidence diagram is inaccessible, leaving its documented query-change rationale unverified; the code itself was verified independently and matches the text claims. View code Optional follow-up: Re-attach or re-host the diagram (e.g. commit it under docs/) so the query-change rationale is reviewable evidence.
    • Ask an agent to verify and fix this finding

7ad9e2c2-770d-4d35-b414-5c7dfc3d2547

Selected findings run after merge and open separate pull requests.

@harlan-zw

Copy link
Copy Markdown
Collaborator Author

🤖 Harlan Agent Kit Agent measured these SQL variants against the live D1 primary. All queries were read-only.

Query Reads before Reads after Time before Time after
Full scoring 13,951,401 165,855 2,438 ms 806 ms
One Skill per owner 74,526 70,154 1,866 ms 86 ms
Embedding backlog 36,693 36,694 2,168 ms 696 ms
Testing tag 121,572 105,778 360 ms 31 ms
Frontend tag 51,645 34,321 184 ms 66 ms
Nuxt tag 42,162 24,388 107 ms 20 ms

These are individual executions, not latency percentiles. Background jobs remained active.

Full scoring returned 20,925 identical rows. Discovery returned 60 identical cards. Backlog counts matched. Tag membership and fields matched across all three samples. Two frontend rows swapped inside an exact stars-and-name tie. The existing order leaves that tie unspecified.

The embedding rewrite reduces text work, rather than index reads. It keeps the same eligibility rules.

@harlan-zw harlan-zw added the harlan-agent-review Pull request triage requires an adversarial Review. Adding this label manually always forces one. label Oct 11, 2026
@harlan-github-agent harlan-github-agent Bot added harlan-agent-ready The automated Review passed every gate on this head commit. and removed harlan-agent-review Pull request triage requires an adversarial Review. Adding this label manually always forces one. harlan-agent-running An Agent holds a Task on this issue or pull request right now. harlan-agent-review-required Pull request triage requires an adversarial Review for this head commit. labels Oct 11, 2026

@harlan-github-agent harlan-github-agent Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Harlan Agent Kit posted this automated review. AI open source policy.

Automated Review: READY.

Read the Review results

@harlan-zw
harlan-zw merged commit 68c1bfc into main Oct 11, 2026
8 checks passed
@harlan-zw
harlan-zw deleted the fix/d1-slow-query-sweep branch October 11, 2026 01:36
@harlan-github-agent harlan-github-agent Bot removed the harlan-agent-ready The automated Review passed every gate on this head commit. label Oct 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant