Skip to content

fix(indexer): re-embed unchanged files when indexing_params change - #301

Merged
georgeh0 merged 1 commit into
mainfrom
reduce/re-embed-files-when-indexing
Oct 6, 2026
Merged

georgeh0 merged 1 commit into
mainfrom
reduce/re-embed-files-when-indexing

Conversation

@georgeh0

@georgeh0 georgeh0 commented Oct 6, 2026

Copy link
Copy Markdown
Member

Summary

  • INDEXING_EMBED_PARAMS is now declared with detect_change=True. Before, editing embedding.indexing_params (e.g. adding prompt_name: passage) and restarting the daemon left files with unchanged content on their memoized process_file results, embedded with the old params. This affected sentence-transformers. LiteLLM was already covered because its constructor kwargs are part of the embedder memo key.
  • QUERY_EMBED_PARAMS stays as is: it is read only at query time, outside any memoized function.
  • Tests: StubEmbedder in conftest now records the kwargs of each embed() call, replacing the duplicate _KwargRecordingEmbedder. The registry index helpers moved from the chunking test into conftest. A new end-to-end test, tests/test_reindex_on_embed_params_change.py, covers the change.

Release note: memo entries written before this change don't record the params dependency. Upgrading from v0.2.42 is still correct, because #299 added a process_file argument and so every file re-runs once. I checked this by indexing with v0.2.42 and then with this branch. This must ship in the same release as #299.

Test plan

  • New test: adding, changing, or removing a param reprocesses both unchanged files, and the embedder sees only the new kwargs. Equal params across a restart, in either key order, leave every file unchanged. The change cases fail without the fix: 0 reprocessed, 2 unchanged.
  • uv run pytest tests/: 329 passed. mypy src/ tests/, ruff check, and ruff format --check are clean.

馃 Generated with Claude Code

INDEXING_EMBED_PARAMS was not a change-detected context key, so editing
embedding.indexing_params (e.g. adding `prompt_name: passage`) and
restarting the daemon left files with unchanged content on their memoized
process_file results, embedded with the old params. LiteLLM was already
covered because its constructor kwargs are part of the embedder memo key;
sentence-transformers was not.

- Declare INDEXING_EMBED_PARAMS with detect_change=True.
  QUERY_EMBED_PARAMS stays as is: it is read only at query time, outside
  any memoized function.
- StubEmbedder in conftest records the kwargs of each embed() call; the
  duplicate _KwargRecordingEmbedder is gone.
- Move the registry index helpers from the chunking test into conftest.
- New end-to-end test: adding, changing, or removing a param reprocesses
  unchanged files with the new kwargs; equal params across a restart
  (in any key order) leave every file unchanged.

Memo entries written before this change don't record the params
dependency. Upgrading from v0.2.42 still re-runs every file once, since
#299 added a process_file argument, so this must ship in the same release.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@georgeh0
georgeh0 merged commit 9feb04f into main Oct 6, 2026
3 of 4 checks passed
@georgeh0
georgeh0 deleted the reduce/re-embed-files-when-indexing branch October 6, 2026 03:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant