fix(local): keep a word's combining marks when the keyword index splits text

docs/adr/0031-ranking-blends-absolute-leg-scores.md

@@ -1,6 +1,6 @@
 # ADR 0031: Ranking blends the two legs on absolute scales, keyword strength being term coverage rather than BM25
 
-- **Status:** Accepted
+- **Status:** Accepted; amended by [ADR 0032](0032-one-tokenizer-for-index-and-question.md), term coverage being safe only where a term is a word
 - **Date:** 2026-09-24
 - **Supersedes:** the ordering decision in [ADR 0006](0006-hybrid-retrieval.md), that the union is ordered by cosine similarity alone and needs neither a stopword list nor rank fusion
 - **Source:** [retrieval eval](../../surfsense_local/backend/scripts/run_retrieval_eval.py), [ADR 0006](0006-hybrid-retrieval.md)

docs/adr/0032-one-tokenizer-for-index-and-question.md

@@ -0,0 +1,27 @@
+# ADR 0032: The index and the question are split by one tokenizer, FTS5's own, and it keeps a word's combining marks
+
+- **Status:** Accepted
+- **Date:** 2026-09-24
+- **Amends:** [ADR 0031](0031-ranking-blends-absolute-leg-scores.md), whose term coverage is safe only where a term is a word
+- **Source:** [retrieval eval](../../surfsense_local/backend/scripts/run_retrieval_eval.py), [ADR 0031](0031-ranking-blends-absolute-leg-scores.md)
+
+## Context
+
+[ADR 0031](0031-ranking-blends-absolute-leg-scores.md) gave the keyword leg 35% of the order and argued no stopword list is needed, because coverage is absolute: a chunk matching one term of five is weak whatever BM25 says. That reasoning holds for a term that is a word.
+
+FTS5's `unicode61` ends a token at every combining mark, so Devanagari `स्कैनर` (scanner) was indexed as `स` + `नर` and `बैटरी` (battery) as `ब` + `टर` — letters, not words. Python's `\w` excludes marks too, so the query was cut the same way and the two happened to agree on nonsense. Nearly every Hindi document holds nearly every Devanagari letter, so coverage stopped discriminating: measured across the eval corpus, the right document covered 96% of a Hindi question's terms and the best wrong document 85%, one pair tying at 91%. A healthy slice is the identifier one, at 86% against 22%. The app ships in [eight languages](../architecture/localization.md), Hindi among them.
+
+## Decision
+
+- **`M*` joins the tokenizer's categories**, so a combining mark stays inside its word. Latin, digits and identifiers tokenise identically either way; this costs them nothing.
+- **One rule splits both the index and the question**, stated in [`shared/tokenizer.py`](../../surfsense_local/backend/shared/tokenizer.py) and asked of FTS5 itself rather than matched by a regex. The question is tokenised by inserting it into a throwaway in-memory FTS5 table declared with the same option and reading the terms back.
+- Rejected: **replicating the rule in Python**. `unicode61` also folds Latin diacritics, so `café` indexes as `cafe`; a regex agreeing with that needs SQLite's own curated table, and every drift is a wrong vote rather than a miss. Measured, fixing only the index side dropped a Hindi question's coverage to 0 of 10 terms.
+- Rejected for CJK: the **`trigram`** tokenizer, which would segment Japanese and Chinese but changes what BM25 means for every other language and grows the index. That gap stays open.
+
+## Consequences
+
+- A Hindi question's coverage margin over the best wrong document goes from +11% to +42%, the wrong document falling from 85% to 27%. Every other slice of the eval is unchanged to the digit, the blend still reaching an answering passage for 98% of 266 queries at mean rank 1.1.
+- The eval's index fingerprint now covers the tokenizer beside the corpus and the embedder. Left out, a tokenizer change scored against an index built under the old rule and reported the old rule's numbers.
+- Splitting a question costs a throwaway SQLite connection, about 355µs against the ~50ms its own embedding costs, once per `retrieve()`. The `ponytail:` in [`shared/tokenizer.py`](../../surfsense_local/backend/shared/tokenizer.py) names a thread-local connection as the fix at 31µs.
+- Migration `0017` rebuilds `chunks_fts`. The index holds no text of its own, so it rebuilds from `chunks` with no re-embed and no download.
+- Bengali, Tamil, Thai and every other script that writes with combining marks gain the same correction, untested here.

docs/adr/README.md

@@ -65,4 +65,5 @@ The Date is the day the decision was first written down. Source links are permal
 | 0028 | [Model-written code may run on the user's machine without a sandbox, and the agent asks before each shell command](0028-model-written-code-runs-with-approval.md) | Accepted |
 | 0029 | [Interface text lives in ICU MessageFormat catalogs, one flat JSON file per language](0029-icu-translation-catalogs.md) | Accepted |
 | 0030 | [FormatJS renders interface text from precompiled, bundled catalogs, with no network at build or run time](0030-formatjs-renders-interface-text.md) | Accepted |
-| 0031 | [Ranking blends the two legs on absolute scales, keyword strength being term coverage rather than BM25](0031-ranking-blends-absolute-leg-scores.md) | Accepted |
+| 0031 | [Ranking blends the two legs on absolute scales, keyword strength being term coverage rather than BM25](0031-ranking-blends-absolute-leg-scores.md) | Accepted, amended by 0032 |
+| 0032 | [The index and the question are split by one tokenizer, FTS5's own, and it keeps a word's combining marks](0032-one-tokenizer-for-index-and-question.md) | Accepted |

docs/architecture/search.md

@@ -2,8 +2,8 @@
 
 `retrieve()` finds the passages in a workspace that best answer a query. Two legs widen recall, a keyword match and a nearest-neighbour search over embeddings, and a weighted blend of the two decides the order, meaning counting for rather more than words. It runs in the calling process against the same SQLite file as everything else, and chat is its only caller.
 
-**Code:** [`shared/search.py`](../../surfsense_local/backend/shared/search.py), [`worker/ingestion/embedding.py`](../../surfsense_local/backend/worker/ingestion/embedding.py)
-**Decisions:** [ADR 0006](../adr/0006-hybrid-retrieval.md), [ADR 0007](../adr/0007-bundled-embeddings.md), [ADR 0031](../adr/0031-ranking-blends-absolute-leg-scores.md)
+**Code:** [`shared/search.py`](../../surfsense_local/backend/shared/search.py), [`shared/tokenizer.py`](../../surfsense_local/backend/shared/tokenizer.py), [`worker/ingestion/embedding.py`](../../surfsense_local/backend/worker/ingestion/embedding.py)
+**Decisions:** [ADR 0006](../adr/0006-hybrid-retrieval.md), [ADR 0007](../adr/0007-bundled-embeddings.md), [ADR 0031](../adr/0031-ranking-blends-absolute-leg-scores.md), [ADR 0032](../adr/0032-one-tokenizer-for-index-and-question.md)
 
 ## Interface
 
@@ -19,11 +19,11 @@ retrieve(session, workspace_id, query, top_k=5, document_ids=None) -> list[Hit]
 ## How it ranks
 
 1. **Embed the query** with the bundled bge-small model at the same width as ingest. The import is lazy, so onnxruntime and the model load on the first query rather than when the API starts.
-2. **Keyword leg.** The query is split into word tokens, lowercased, each quoted against FTS5's query grammar and joined with `OR` to keep recall wide. `chunks_fts` is matched, joined to `chunks` and `documents` for the workspace and document filters, and the best 20 by BM25 are kept. Each candidate then scores the fraction of the query's distinct terms it matched, one further lookup per term.
+2. **Keyword leg.** The query is split into terms by the index's own tokenizer, each quoted against FTS5's query grammar and joined with `OR` to keep recall wide. `chunks_fts` is matched, joined to `chunks` and `documents` for the workspace and document filters, and the best 20 by BM25 are kept. Each candidate then scores the fraction of the query's distinct terms it matched, one further lookup per term.
 3. **Vector leg.** A sqlite-vec nearest-neighbour search over `chunk_vectors` takes the 20 closest chunks. It sits in its own CTE, so only `MATCH` and `k` constrain it, as vec0 requires; the workspace and document filters then apply to that set. Each candidate scores its cosine similarity, floored at zero.
 4. **Blend.** `0.65 × semantic + 0.35 × keyword`, a leg a candidate is missing from contributing nothing. The union is cut to `top_k`, cosine breaking a tie.
 
-Both legs decide the order, and both score on a scale that means the same thing from one query to the next, so a leg with nothing to say adds nothing. That is what keeps a keyword match honest: a chunk matching one term of five is weak whatever BM25 says about it, so no stopword list is needed. A paraphrase that shares no words with its passage still arrives through the vector leg. There is no reranker.
+Both legs decide the order, and both score on a scale that means the same thing from one query to the next, so a leg with nothing to say adds nothing. That is what keeps a keyword match honest: a chunk matching one term of five is weak whatever BM25 says about it, so no stopword list is needed. It holds because a term is a word: [`shared/tokenizer.py`](../../surfsense_local/backend/shared/tokenizer.py) states the one rule that splits both the index and the question, and keeps a combining mark inside its word so Devanagari is not cut into letters ([ADR 0032](../adr/0032-one-tokenizer-for-index-and-question.md)). A paraphrase that shares no words with its passage still arrives through the vector leg. There is no reranker.
 
 `Hit.score` is cosine similarity, so it says how close a passage is, not where it sits.
 
@@ -33,15 +33,15 @@ The vector leg looks at the 20 nearest chunks across every workspace before it f
 
 ## Index
 
-`chunks_fts` is an FTS5 table with external content over `chunks`, and `chunk_vectors` a vec0 table keyed by chunk id. Triggers on `chunks` keep the keyword index in step, including on cascade, and ingest writes the vectors ([`data-model.md`](data-model.md), [`documents.md`](documents.md)). Because both indexes key on chunk ids, a hit is a chunk row, and the citation panel can load that chunk's neighbours by position.
+`chunks_fts` is an FTS5 table with external content over `chunks`, declared with the tokenizer `shared/tokenizer.py` names, and `chunk_vectors` a vec0 table keyed by chunk id. Triggers on `chunks` keep the keyword index in step, including on cascade, and ingest writes the vectors ([`data-model.md`](data-model.md), [`documents.md`](documents.md)). Because both indexes key on chunk ids, a hit is a chunk row, and the citation panel can load that chunk's neighbours by position.
 
 ## Tests
 
-[`tests/integration/search/test_retrieve.py`](../../surfsense_local/backend/tests/integration/search/test_retrieve.py) covers a keyword query, a paraphrase found through meaning, the document and lines on a hit, scoping to selected documents and to the workspace, and the empty workspace, query and selection. [`tests/integration/chunks/test_search_index.py`](../../surfsense_local/backend/tests/integration/chunks/test_search_index.py) covers the triggers and the refusal of a vector of the wrong width.
+[`tests/integration/search/test_retrieve.py`](../../surfsense_local/backend/tests/integration/search/test_retrieve.py) covers a keyword query, a paraphrase found through meaning, a Hindi question against three notes one word apart, the document and lines on a hit, scoping to selected documents and to the workspace, and the empty workspace, query and selection. [`tests/integration/chunks/test_search_index.py`](../../surfsense_local/backend/tests/integration/chunks/test_search_index.py) covers the triggers, the refusal of a vector of the wrong width, and that the index and a question split text the same way — the migration and [`shared/tokenizer.py`](../../surfsense_local/backend/shared/tokenizer.py) state the tokenizer separately, and a drift between them scores a question against terms the index never held.
 
 Ranking itself is measured rather than asserted, by [`scripts/run_retrieval_eval.py`](../../surfsense_local/backend/scripts/run_retrieval_eval.py): it indexes a fixed corpus through the real ingest pipeline and records where each query's answering passage landed. Failures that need a library-sized corpus, a bare identifier among near-duplicate manuals being the one that drove [ADR 0031](../adr/0031-ranking-blends-absolute-leg-scores.md), do not reproduce at the handful of documents an integration test builds.
 
 ## Known gaps
 
 - A question in one language does not find its answer in another. Cosine alone puts the answering passage in the top 5 for 1 of 8 such queries, and the blend for 2 of 8: bge-small is English-only, so no ranking recovers it. A multilingual embedder is the fix.
-- FTS5's tokenizer keeps a Japanese or Chinese clause as one token, so the keyword leg finds nothing in those scripts and the blend runs on meaning alone there.
+- FTS5's tokenizer keeps a Japanese or Chinese clause as one token, so the keyword leg finds nothing in those scripts and the blend runs on meaning alone there. `trigram` would segment them at the cost of what BM25 means everywhere else ([ADR 0032](../adr/0032-one-tokenizer-for-index-and-question.md)).

surfsense_local/backend/alembic/versions/0017_keyword_index_keeps_combining_marks.py

@@ -0,0 +1,45 @@
+"""Keep a word's combining marks inside it when the keyword index splits text.
+
+unicode61 ends a token at every combining mark unless `M*` is among its
+categories, so Devanagari `स्कैनर` (scanner) was indexed as `स` + `नर`: letters,
+not words. Since ADR 0031 the keyword leg weighs what fraction of a question a
+chunk matched, and nearly every Hindi document holds nearly every Devanagari
+letter, so those fragments did not merely fail to match, they voted.
+
+The literal below is frozen, as a migration's is; `shared/tokenizer.py` states
+the live rule and an integration test holds the two together. The index keeps
+no text of its own, so this rebuilds from `chunks` and needs no re-embed.
+
+Revision ID: 0017
+Revises: 0016
+
+"""
+
+from collections.abc import Sequence
+
+from alembic import op
+
+revision: str = "0017"
+down_revision: str | Sequence[str] | None = "0016"
+branch_labels: str | Sequence[str] | None = None
+depends_on: str | Sequence[str] | None = None
+
+TOKENIZER = "unicode61 categories 'L* N* Co M*'"
+
+
+def _rebuild(tokenize: str | None) -> None:
+    option = f', tokenize="{tokenize}"' if tokenize else ""
+    op.execute("DROP TABLE chunks_fts")
+    op.execute(
+        "CREATE VIRTUAL TABLE chunks_fts USING fts5("
+        f"content, content='chunks', content_rowid='id'{option})"
+    )
+    op.execute("INSERT INTO chunks_fts(chunks_fts) VALUES('rebuild')")
+
+
+def upgrade() -> None:
+    _rebuild(TOKENIZER)
+
+
+def downgrade() -> None:
+    _rebuild(None)

surfsense_local/backend/scripts/retrieval_eval/fingerprint.py

@@ -9,15 +9,21 @@
 from pathlib import Path
 
 
-def fingerprint(*corpus_dirs: Path, embedder: bytes) -> str:
-    """Every document that will be indexed, plus the model that will embed it."""
+def fingerprint(*corpus_dirs: Path, embedder: bytes, tokenizer: str) -> str:
+    """Every document that will be indexed, and the two rules that turn it into one.
+
+    The tokenizer belongs here with the embedder: it decides the terms the
+    keyword leg scores, so changing it changes the index as surely as a new
+    model does.
+    """
     digest = hashlib.sha256()
     for corpus_dir in corpus_dirs:
         for path in sorted(corpus_dir.glob("*.md")) if corpus_dir.is_dir() else []:
             digest.update(path.name.encode("utf-8"))
             digest.update(path.read_bytes())
             digest.update(b"\0")
     digest.update(embedder)
+    digest.update(tokenizer.encode("utf-8"))
     return digest.hexdigest()[:16]
 
 

surfsense_local/backend/scripts/run_retrieval_eval.py

@@ -19,6 +19,9 @@
 from retrieval_eval.cases import CORPUS_DIR, LOCAL_DIR
 from retrieval_eval.fingerprint import embedder_identity, fingerprint
 
+# Safe this high, unlike the embedder below: it reads no settings, only sqlite3.
+from shared.tokenizer import TOKENIZER
+
 # Deliberately not imported from worker.ingestion.embedding, even though that
 # is where it is defined: importing anything under `worker` pulls in
 # shared.queue, which reads the settings at import time and caches them. That
@@ -40,14 +43,15 @@
     str(_STAGED if (_STAGED / "bge-small-en-v1.5").is_dir() else _DATA / "models"),
 )
 # Keyed by what was indexed, so a ranking change reuses the index and only a
-# corpus or embedder change pays to build one.
+# corpus, embedder or tokenizer change pays to build one.
 CACHE_ROOT = Path.home() / ".surfsense-retrieval-eval"
 _KEY = fingerprint(
     CORPUS_DIR,
     LOCAL_DIR / "corpus",
     embedder=embedder_identity(
         Path(os.environ["SURFSENSE_LOCAL_MODELS_DIR"]), MODEL_DIR_NAME
     ),
+    tokenizer=TOKENIZER,
 )
 os.environ["SURFSENSE_LOCAL_DATA_DIR"] = str(CACHE_ROOT / _KEY)
 

surfsense_local/backend/shared/search.py

@@ -1,14 +1,14 @@
-import re
 from collections.abc import Sequence
 from dataclasses import dataclass
 
 from sqlalchemy import bindparam, text
 from sqlalchemy.orm import Session
 
+from shared.tokenizer import terms
+
 # Each leg proposes this many for recall; the blend below orders the union.
 CANDIDATES = 20
 
-_WORD = re.compile(r"\w+")
 _COLUMNS = "c.id, c.document_id, d.title, c.content, c.start_line, c.end_line"
 
 
@@ -96,13 +96,19 @@ def _keyword_leg(
     query: str,
     document_ids: Sequence[int] | None,
 ) -> list[tuple[int, float]]:
-    terms = _WORD.findall(query.lower())
-    if not terms:
+    # Split by the index's own rule, or the question asks for terms it never
+    # held: `स्कैनर` cut at its virama is `स` + `नर`, and this leg then abstains
+    # on every Devanagari question.
+    distinct = terms(query)
+    if not distinct:
         return []
 
     # Quote each term against FTS5's grammar; OR keeps recall wide.
     candidates = _matching(
-        session, workspace_id, " OR ".join(f'"{term}"' for term in terms), document_ids
+        session,
+        workspace_id,
+        " OR ".join(f'"{term}"' for term in distinct),
+        document_ids,
     )
     if not candidates:
         return []
@@ -114,13 +120,12 @@ def _keyword_leg(
     # ponytail: one small indexed lookup per distinct term, so a long question
     # costs a few more. Move to an fts5vocab table if queries ever get long.
     covered = dict.fromkeys(candidates, 0)
-    for term in set(terms):
+    for term in distinct:
         for chunk_id in _matching(
             session, workspace_id, f'"{term}"', document_ids, within=candidates
         ):
             covered[chunk_id] += 1
-    distinct = len(set(terms))
-    return [(chunk_id, count / distinct) for chunk_id, count in covered.items()]
+    return [(chunk_id, count / len(distinct)) for chunk_id, count in covered.items()]
 
 
 def _matching(

surfsense_local/backend/shared/tokenizer.py

@@ -0,0 +1,33 @@
+"""The one rule by which text becomes terms, for the index and the question alike.
+
+A term the index keeps whole and a question cuts up matches nothing. Since
+[ADR 0031](../../../docs/adr/0031-ranking-blends-absolute-leg-scores.md) the
+keyword leg scores what fraction of a question a chunk matched, so a
+disagreement no longer merely misses: it votes. FTS5 itself is asked rather
+than matched by a regex, which would have to track its case folding and its
+Latin diacritic table to stay honest.
+"""
+
+import sqlite3
+
+# `M*` keeps a combining mark inside its word. Without it unicode61 ends a token
+# at every mark, so Devanagari `स्कैनर` indexes as `स` + `नर`: letters, not
+# words, and nearly every Hindi document holds nearly every letter. Latin,
+# digits and identifiers tokenise the same either way.
+TOKENIZER = "unicode61 categories 'L* N* Co M*'"
+
+
+def terms(text: str) -> list[str]:
+    """The distinct terms FTS5 indexes for `text`, sorted.
+
+    Distinct because coverage asks what fraction of a question a chunk matched,
+    which a word repeated in the question must not inflate.
+    """
+    # ponytail: a throwaway connection per call, ~355us against the ~50ms the
+    # query's own embedding costs, and retrieve() calls this once. Hold one in a
+    # threading.local (~31us) if it ever runs in a loop.
+    db = sqlite3.connect(":memory:")
+    db.execute(f'CREATE VIRTUAL TABLE q USING fts5(body, tokenize="{TOKENIZER}")')
+    db.execute("CREATE VIRTUAL TABLE v USING fts5vocab(q, row)")
+    db.execute("INSERT INTO q(body) VALUES (?)", (text,))
+    return [term for (term,) in db.execute("SELECT term FROM v")]