Repository navigation
Decouple index record models from LanceModel / Vector #476
Description
The seven derived-index record models (everos.infra.persistence.index tables) subclass lancedb.pydantic.LanceModel and type their vector columns with lancedb.pydantic.Vector(dim). Milvus consumes them only through the backend-neutral IndexSchema (schema_for), so the dependency is one-directional, but the models are still named and typed by one backend's library.
Scope: replace LanceModel with a plain pydantic base and Vector(dim) with a neutral annotation; the LanceDB adapter then owns the pydantic→Arrow mapping (pydantic_to_schema special-cases lancedb's FixedSizeListMixin, so a custom mapper is required).
Deliberately left out of the (app_id, project_id, id) primary-key work: it touches all seven tables, six test files and the Arrow conversion, with no behaviour change. Tracked so it is not forgotten.
Activity
gloryfromca commentedon Oct 20, 2026
#510 made the cost of this coupling concrete, so recording it here while it is fresh.
It is not a storage-engine constraint. Both backends take the vector width as an ordinary runtime argument when the table is created — Milvus passes
dim=field.dimensionwhile building the collection schema, and a LanceDB table is created from an Arrow schema whosefixed_size_listsize is just a number. Nothing underneath requires the width to be known early.What requires it is that the schema is declared as a Python class.
vector: Vector(_DIM)is a type annotation, so it is evaluated when the class body runs — at import time, before anything has read a config file or parsed a command line. The width is written into the class's definition rather than handed over when a table is built.Making
dimconfigurable therefore had to drag configuration-reading back to import time, which is where these came from:_bootstrap_root_from_argv()inentrypoints/cli/main.py--rootlong after the record classes exist, so argv has to be pre-scannedinfra/persistence/records/_dim.pySettingscannot be built at import time — a malformedeveros.tomlwould breakeveros config show, the command used to repair itassert_record_dim_matches_settings()ininfra/persistence/index/__init__.pyA review of #510 also found a defect that lives entirely in the first of those: with both a group-level and a command-level
--root, the prescan and typer picked different ones. Fixed there, but the whole class of bug disappears with this issue.Verification for whoever picks this up: the change is supposed to be invisible, which is hard to demonstrate by testing behaviour. Dump every business table's
to_arrow_schema()before the change and compare field by field afterwards — name, type and nullability for all seven tables. That comparison is the acceptance criterion; a behavioural test suite passing proves much less here.One thing worth folding in while the mapping is being rewritten: lancedb's
_pydantic_to_field(name, field)returnspa.field(name, dt, nullable)with no metadata parameter, so a table created from aLanceModelcannot carry Arrow field metadata. That is whycascade rebuildhas to write the embedding-model record explicitly after rebuilding, instead of the table carrying it from birth. Owning the mapping removes that special case too.Priority: low. No behaviour changes, nothing is broken today, and the workarounds above are small and tested. This is cleanup that pays off the next time something needs to be known before the record classes are created.
feat!: the everalgo 2026-10 line and the target architecture (EverMin…