[MODSetter]

What

FTS5's unicode61 ends a token at every combining mark, so Devanagari स्कैनर (scanner) was indexed as स + नर and बैटरी (battery) as ब + टर — letters, not words. Python's \w drops marks too, so the question was cut the same way and the two sides agreed on nonsense.

• M* joins the tokenizer's categories. Latin, digits and identifiers tokenise identically either way, so this costs them nothing.
• One rule splits both the index and the question, stated in shared/tokenizer.py. The question is tokenised by FTS5 itself — inserted into a throwaway in-memory table declared with the same option — rather than matched by a regex.
• Migration 0017 rebuilds chunks_fts. The index holds no text of its own, so it rebuilds from chunks: no re-embed, no download.
• The eval's index fingerprint now covers the tokenizer beside the corpus and the embedder (second commit, separable).

ADR 0032 records it and amends ADR 0031; docs/architecture/search.md follows.

Why

ADR 0031 gave the keyword leg 35% of the order and argued no stopword list is needed, because coverage is absolute: a chunk matching one term of five is weak whatever BM25 says. That holds where a term is a word. Nearly every Hindi document holds nearly every Devanagari letter, so coverage stopped discriminating.

Measured across the eval corpus — every document scored against every query the way _keyword_leg does, right document against best wrong document:

Slice	before	after
hindi	96% / 85% (+11%)	69% / 27% (+42%)
identifier	86% / 22% (+63%)	unchanged
english, paraphrase, superseded, near-duplicate, german, japanese, chinese, cross-lingual		unchanged to the digit

One Hindi pair tied at 91% against 91%. This was harmless before ADR 0031, when a keyword match only widened recall and could not reorder; it votes now, and Hindi is one of the eight languages the app ships in.

The eval's own total does not move — 98% of 266 queries at mean rank 1.1, before and after. The Hindi slice was already 100% on five queries carried by the vector leg, so a top-5 rate cannot show a margin. The table above is the result; the eval is the check that nothing else regressed.

Two things ruled out by measurement:

• Fixing only the index. Drops a Hindi question to 0 of 10 terms matched, because the query splitter still cuts at marks. Both sides had to move together, which is why there is one module rather than two constants.
• Replicating the rule in Python. unicode61 also folds Latin diacritics, so café indexes as cafe; a regex agreeing with that needs SQLite's own curated table, and every drift is a wrong vote rather than a miss.

Still open, unchanged by this: an entire Japanese or Chinese clause is one token, so the keyword leg scores 0% there. trigram is the only tokenizer that segments it and it changes what BM25 means for every other language. Cross-lingual retrieval stays at 25% — that is the embedder, not the ranking. Both are Known gaps in search.md.

Fixes #

How to test

cd surfsense_local/backend
uv run pytest -m unit -k tokenizer            # the rule itself
uv run pytest tests/integration/search tests/integration/chunks

Two guards are worth reading rather than just running:

• test_a_hindi_question_finds_the_note_that_answers_it — three notes one word apart, so meaning leaves them near-tied and only the keyword leg can choose. Revert shared/tokenizer.py's M* and it asks about the scanner and returns the printer.
• test_the_index_splits_text_the_way_a_question_will — the migration and shared/tokenizer.py state the tokenizer separately, so editing one without the other fails here.

The full measurement, against a rebuilt index:

uv run scripts/run_retrieval_eval.py run --out /tmp/marks.jsonl

Existing libraries upgrade on next launch; alembic upgrade head rebuilds the index in place.

High-level PR Summary

This PR fixes a critical tokenization issue where FTS5's unicode61 tokenizer was breaking Devanagari (and other combining-mark scripts) words into individual letters instead of keeping them whole. By adding M* (combining marks) to the tokenizer categories and ensuring both the search index and query use the same tokenization rule, Hindi queries now properly match Hindi documents. The fix significantly improves keyword matching for Hindi from +11% margin to +42% margin over wrong documents, while maintaining identical performance for other languages. A database migration rebuilds the keyword index with the new tokenizer, and the eval fingerprint now includes the tokenizer alongside the corpus and embedder to detect configuration changes.

⏱️ Estimated Review Time: 30-90 minutes

💡 Review Order Suggestion

Order	File Path
1	docs/adr/0032-one-tokenizer-for-index-and-question.md
2	shared/tokenizer.py
3	surfsense_local/backend/tests/unit/shared/test_tokenizer.py
4	surfsense_local/backend/shared/search.py
5	surfsense_local/backend/alembic/versions/0017_keyword_index_keeps_combining_marks.py
6	surfsense_local/backend/tests/integration/chunks/test_search_index.py
7	surfsense_local/backend/tests/integration/search/test_retrieve.py
8	surfsense_local/backend/scripts/retrieval_eval/fingerprint.py
9	surfsense_local/backend/scripts/run_retrieval_eval.py
10	surfsense_local/backend/tests/unit/scripts/test_retrieval_eval_fingerprint.py
11	docs/adr/0031-ranking-blends-absolute-leg-scores.md
12	docs/adr/README.md
13	docs/architecture/search.md

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
surf-sense-frontend	Ready	Preview	Sep 25, 2026 5:59am UTC

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 07bfb52f-1b6c-4ccc-b9a1-734ea6f3a74b

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}