Repository navigation
fix(indexer): re-embed unchanged files when indexing_params change - #301
Merged
Merged
Conversation
INDEXING_EMBED_PARAMS was not a change-detected context key, so editing embedding.indexing_params (e.g. adding `prompt_name: passage`) and restarting the daemon left files with unchanged content on their memoized process_file results, embedded with the old params. LiteLLM was already covered because its constructor kwargs are part of the embedder memo key; sentence-transformers was not. - Declare INDEXING_EMBED_PARAMS with detect_change=True. QUERY_EMBED_PARAMS stays as is: it is read only at query time, outside any memoized function. - StubEmbedder in conftest records the kwargs of each embed() call; the duplicate _KwargRecordingEmbedder is gone. - Move the registry index helpers from the chunking test into conftest. - New end-to-end test: adding, changing, or removing a param reprocesses unchanged files with the new kwargs; equal params across a restart (in any key order) leave every file unchanged. Memo entries written before this change don't record the params dependency. Upgrading from v0.2.42 still re-runs every file once, since #299 added a process_file argument, so this must ship in the same release. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
INDEXING_EMBED_PARAMSis now declared withdetect_change=True. Before, editingembedding.indexing_params(e.g. addingprompt_name: passage) and restarting the daemon left files with unchanged content on their memoizedprocess_fileresults, embedded with the old params. This affected sentence-transformers. LiteLLM was already covered because its constructor kwargs are part of the embedder memo key.QUERY_EMBED_PARAMSstays as is: it is read only at query time, outside any memoized function.StubEmbedderin conftest now records the kwargs of eachembed()call, replacing the duplicate_KwargRecordingEmbedder. The registry index helpers moved from the chunking test into conftest. A new end-to-end test,tests/test_reindex_on_embed_params_change.py, covers the change.Release note: memo entries written before this change don't record the params dependency. Upgrading from v0.2.42 is still correct, because #299 added a
process_fileargument and so every file re-runs once. I checked this by indexing with v0.2.42 and then with this branch. This must ship in the same release as #299.Test plan
uv run pytest tests/: 329 passed.mypy src/ tests/,ruff check, andruff format --checkare clean.馃 Generated with Claude Code