Repository navigation
feat: v0.2 ranked search by words and meaning, and fetchle -i - #12
Open
codywilliamson wants to merge 65 commits into
Open
codywilliamson wants to merge 65 commits into
codywilliamson wants to merge 65 commits into
Conversation
Bench numbers follow in a docs commit once the quiet-runner after-run finishes.
IndexBenchmarks.Search, quiet linux runner, JIT, MemoryDiagnoser. before = bench.yml run 39628557808 (main 087bb80), after = run 39629887529 (9870e41). mean / allocated: 50k .json 633.2 us / 7400 B -> 631.4 us / 7400 B 50k settings 170.2 us / 1536 B -> 148.1 us / 1536 B 50k zzz 159.4 us / 712 B -> 139.1 us / 712 B 1M .json 17,054.8 us / 7400 B -> 16,026.2 us / 7400 B 1M settings 3,828.9 us / 10,216 B -> 3,428.1 us / 10,216 B 1M zzz 3,642.3 us / 712 B -> 3,168.2 us / 712 B Windows runner could not load the benchmark assemblies in either run (Application Control policy), so no windows numbers.
…n keys snapshot format 2 adds seven sections built per unique segment at rebuild time, all bounds-checked on open. a v1 snapshot reads as unreadable (exit 3, rebuild).
226 mined queries plus the 12 spike queries over a 1M-entry in-memory corpus: the seeded home corpus plus 15,386 real (pseudonymized) paths from transcript listings. ids hash into dev and held-out; the gate and baseline read only held-out. v0.1 substring on dev: top-1 12.6%, top-3 15.6%, mrr 0.142.
terms expand through the lexicon (exact, prefix by binary search, edit distance 1 by deletion keys), score unique segments once, then one forward pass carries per-term ancestor matches down the tree into a top-100. dev: top-1 64.7%, top-3 80.2%, mrr 0.738 (v0.1 12.6 / 15.6 / 0.142).
segments embed once into 64-byte sign codes of their centered vector; each entry stores its path centroid length. a query scores every segment through 64 lookup tables, then the entry pass turns a running sum down the path into the centroid cosine. vectors.fvec is validated like the snapshot and bound to its checksum; stale or broken vectors turn semantic off, never fail a search.
a root the snapshot doesn't cover is walked into an in-memory snapshot and ranked by the same pass; fusion merges each retriever's candidates across sources by score, ties share a rank and break on path. the v0.1 substring pass, segment matcher and hit collector are gone.
the snapshot lands first so lexical answers from the new tree while segments embed; vectors.fvec swaps in the same temp, flush, rename way and names the snapshot it belongs to.
the biggest dev miss class: the same file under .claude/worktrees or *-worktrees sat shallower than the real checkout and won on depth. dev: top-1 67.5 -> 72.8%, top-3 79.8 -> 83.8%, mrr 0.747 -> 0.785. penalty 0.3/0.5/0.7 measured 72.4/72.8/72.8% top-1, kept 0.5.
dev: top-1 72.8 -> 74.6%, top-3 83.8 -> 85.1%, mrr 0.785 -> 0.797. copy marks are a per-segment bitset so the entry pass tests a bit, not a hash.
semantic alone on dev was 42.5% top-1 and plain rrf cut fused top-1 from 74.6% (lexical) to 63.2%: one-word dirs like .vscode are their whole centroid and the depth term favored them. now only files are semantic candidates, no semantic depth term, semantic ranks count 0.75, and nothing when some file matched every query word. sweeps (semantic weight 0.4-1, covered 0-1, min cosine .35/.45, files-only, depth .02/0/-.01, numbers in embed text, coverage-weighted rrf) are in docs/decisions.md. dev: 77.2 / 88.2 / 0.831. spike s-06 and s-11 now accept both profile copies the corpus holds.
… doctor search opens the model with bounds checks only (one 33 MB checksum pass per query would eat the latency budget); rebuild and doctor verify the checksum. no model means lexical ranking and a doctor warning, never a failure.
…n the bench runners the external bench keeps the cold rebuild row snapshot-only (v0.1's 1.2x target) and adds a rebuild-with-vectors row. the picker bench shares the 1m home index.
when no path matched any query word, the best semantic hit must reach cosine 0.5 or the result is empty (exit 1). a per-candidate floor cost 2 points of dev top-3; this one costs nothing on dev. e2e assertions move from exact sets to ranked expectations: the target first, excluded paths never.
…che sibling scores rows are breadth-first, so a depth level is one contiguous range whose parents all sit in the level above; big levels split across workers with their own heaps, merged by the heaps' total order. siblings whose own name matched nothing share one lexical score, computed once per parent. the loop no longer re-slices snapshot spans per row. dev eval unchanged (77.2 / 88.2 / 0.831). quiet-runner numbers: see the next commit (bench before/after)
…rd counts as covered two classes from the held-out failure table, both checked on dev before held-out was read again. folder/directory/dir words become intent, not terms: files score half. and a dir naming every query word now switches semantic off like a file would; without it, semantic files under the folder outvoted the folder itself. dev: 77.2 / 88.2 / 0.831 -> 78.1 / 89.0 / 0.840. held-out: 75.3 / 90.1 / 0.832 -> 79.0 / 93.8 / 0.864.
baseline: held-out top-1 79.0%, top-3 93.8%, mrr 0.864 over 81 queries. a drop in any of the three fails ci; checked by raising the baseline (exit 1).
…them install.ps1 and install.sh copied only the exe, so an installed fetchle ranked by words only. the lifecycle scripts now assert the model is in place.
on enter or esc the session was disposed while the worker could still be inside the entry pass, reading unmapped pages: an access violation aot can't catch. LatestWinsSearch.Stop cancels, drops what's waiting and blocks until idle, and the token now reaches the entry pass, checked per chunk beside the deadline. found by the independent review.
…st word a megabyte-long word stackalloc'd its length in SplitParts and could kill the mcp server with a stack overflow. the picker's typing word could also land on an earlier term after the 8-term cap or a repeat. found by the independent review.
…idth models count as none from the independent review: a covered query still padded results with semantic candidates scored 0; a file named worktree.ts or notes.wt was discounted as a copy; covered ignored --since; a 256-dim model threw ArgumentException deep in search; tied candidates at the 100 cutoff depended on an unstable sort; paths deeper than 256 got a truncated centroid norm; a query equal to the model mean gave NaN. dev 78.1 / 89.0 / 0.840 -> 77.6 / 89.0 / 0.837, held-out unchanged at 79.0 / 93.8 / 0.864.
covered meant every word the lexicon knew; a word missing from the whole index dropped out, so the rest could cover the query and switch semantic off exactly when it was needed. caught by the new native e2e: "music streaming app" over a tree with no music anywhere missed Spotify.exe. eval unchanged (its corpus has music). e2e now covers the model beside the published exe, vectors on rebuild, a meaning-only hit, lexical-only without the model, and an empty result.
two of three full runs timed out a picker test after 13-16 s while parallel tests held the pool; five of five pass with 64 minimum threads.
ubuntu-latest moved to glibc 2.43; the exe it built failed on the quiet linux runner with GLIBC_2.43 not found, and a release built there would fail on most current distros the same way.
one xxhash3 stream over a 30 MB mapping is bound by page faults on one thread, and every cli query pays it twice. the checksum is now xxhash3 of the chunks' xxhash3s, chunks hashed on every core. snapshot format 3, vectors format 2. the model keeps its single-stream checksum: only rebuild and doctor verify it. quiet-runner numbers: in the commit that records the bench after this one.
the segment matcher and substring pass #11 fixed are gone here, replaced by the ranked pass, whose deadline is checked per 16k-row chunk.
…sibling a case-sensitive volume can hold both Foo and FoO, so a flipped spelling that exists proved nothing and could merge two distinct roots. pathconf with _PC_CASE_SENSITIVE answers for the volume itself; unknown keeps the case-insensitive default.
the word being typed ("ter" of terminal) embeds as noise, so meaning reads only
the finished words, and the session keeps the per-segment scores for the last
text it embedded. most keystrokes extend a word, so most skip the ~4 ms scoring
pass over every segment. cli searches (no prefix) are unchanged.
quiet-runner numbers: in the commit that records the bench after this one.
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
a rebuild with the model leaves vectors.fvec in the index dir, which uninstall didn't know, so the data dir survived (lifecycle failed on all three os).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
fetchle now ranks: a query is words, not a substring, and with the model beside the exe it also matches by meaning. "music streaming app" finds Spotify.exe; "rules for skipping folders during the fetchle walk" finds PruneRules.cs.
This builds on #11 (merged into this branch; merge #11 first and this diff shrinks to the ranking work).
How it works
Read docs/architecture.md, "Words in names" through "The picker", or the five diagrams in docs/diagrams (tokenizer, lexical-index, vector-store, query-flow, picker).
potion-retrieval-32M.fmodelshipped beside the exe (parity with Python: min cosine 0.9975). Every unique name embeds once at rebuild into 64 bytes of sign bits invectors.fvec.fetchle -iis the live picker.Eval
318 judged queries: 226 mined from real file hunts in Claude Code transcripts, 80 written blind from the corpus, the spike's 12. 1M-entry corpus built in memory. Held-out split only for the gate.
Lexical alone scores 81.5 / 93.8 / 0.875 on the same queries: semantic finds what words can't and costs a couple of first places. Every miss and its stage: evals/failures.md.
Latency (quiet runners, published AOT exe)
Pending: runs 39650885254 (the level-parallel pass alone) and 39652918353 (this branch). The run before the perf work (39644601983, Windows, Ryzen 5 3600) measured:
This table gets the after numbers before merge.
Reviews
Not verified
fetchle index rebuildwith this build so the snapshot is format 3 and vectors exist)