[Windsander]

Motivation

Different embedding models produce vectors with different dimensions (e.g., intfloat/multilingual-e5-small uses 384, while intfloat/multilingual-e5-large uses 1024). Currently EverOS hard-codes _DIM = 1024 in LanceDB table schemas and build_embedding_provider, which prevents using smaller/larger embedding models without patching source code.

Changes

• Added dim field to EmbeddingSettings with env binding EVEROS_EMBEDDING__DIM.
• Added dim = 1024 default in default.toml.
• Updated build_embedding_provider to use settings.dim when no explicit dim is passed.
• Updated LanceDB table schemas (agent_case, agent_skill, atomic_fact, episode, foresight, knowledge_topic) to read _DIM from load_settings().embedding.dim instead of hard-coding 1024.

Backward compatibility

The default remains 1024, so existing configurations continue to work without changes.

Testing

• python -m py_compile passes for all modified Python files.
• The changes are minimal and do not alter behavior unless EVEROS_EMBEDDING__DIM or [embedding].dim is explicitly set.

[Windsander]

Hi EverOS team, thanks for the great work on this project.

We're building CodeKeeper Advance (which will release to Github after we fullfill all features 😊) on top of EverOS. In our setup, we use embedding models like intfloat/multilingual-e5-small, which outputs 384-dim vectors. The hard-coded _DIM = 1024 in LanceDB table schemas and build_embedding_provider currently prevents us from using these models without forking/patching EverOS.

This PR makes the embedding dimension configurable via EVEROS_EMBEDDING__DIM or [embedding].dim, while keeping 1024 as the default so existing users are not affected. The changes are intentionally minimal and only touch the embedding pipeline.

Happy to adjust naming or implementation based on your feedback. Thanks for reviewing!

[gloryfromca]

Closing this, but the diagnosis was right and it was right two months before anyone filed it as a bug.

_DIM = 1024 hard-coded in the table schemas is wrong, and you spotted it from the other direction — wanting a 384-dim model rather than hitting a failure. #434 later reported the failure mode: a 768-dim provider (Ollama's nomic-embed-text, the most common local default) is accepted by every surface a user would check — /health says embed: true, the write returns 202, search returns 200 — and fails asynchronously in the cascade with rows marked retryable=FALSE. Silent, non-retryable, with a green health check.

The reason I'm closing rather than rebasing is structural: six of this PR's nine files no longer exist at those paths. #481 (1805535) moved infra/persistence/lancedb/tables/ to infra/persistence/records/ when it re-keyed rows on (app_id, project_id, id). What survives a rebase is the config half — the dim field, its env binding, the default.toml entry, and the build_embedding_provider default — about 12 lines. The six schema files would be rewritten against a different tree, which is not a rebase.

Two other things we want that this PR doesn't carry, and that change the shape of the work:

Configurable dim does not fix the silent failure. Set dim wrong, or don't know it exists, and the symptom is identical. What #434 actually needs is a startup probe: embed once, compare the real vector length against the configured dimension, and refuse to start when the model produces fewer dimensions than configured — a short vector cannot be padded into a valid long one. Over-length is the opposite case and stays as it is today: truncate, on the assumption that a model you deliberately truncate is a Matryoshka one, with a startup warning so the user knows it is happening.

Changing dim on an existing index needs a story, and there is none today. Nothing records which model or dimension produced the stored vectors — I checked, there is no provenance anywhere in the index. Swap the embedding model and the old vectors silently become incomparable; change dim and their shape no longer matches the schema. The probe above is the natural place to catch both: record {model, dim} beside the index, compare at startup, and refuse with a pointer to everos cascade rebuild when it moved.

So the plan is a probe plus provenance first — independent of the schema, and on its own it turns #434 from "installed it, nothing is searchable, everything says OK" into one clear error at startup — then the configurable dimension on top of it, with the schema tests this PR's py_compile could not provide.

Thanks for the report embedded in this PR; the motivation section is what made the connection to #434 obvious once that landed.

[gloryfromca]

The plan described when this was closed has landed, in #510.

[embedding] dim (env EVEROS_EMBEDDING__DIM) now sets the width of every index vector column, so intfloat/multilingual-e5-small runs with dim = 384 and no source patch — that configuration is covered end to end: the column is built at 384, rows are written, and method=vector search returns them.

The probe and provenance half went in underneath it, as described. Startup embeds once, measures the model's real output width, and compares it against both the configured dim and the model recorded on the index. A model too narrow for the column, or a model swap that would make the stored vectors incomparable, now stops the start with one message naming the values and the command that fixes it, instead of failing later in the cascade with no error anywhere a user would look.

What this means for an existing deployment:

• Leaving dim alone needs no action. The first start records which model built the index, in place, without touching a row or a column type.
• Changing dim on an existing index stops the next start: vectors of different lengths have nothing to migrate, so everos cascade rebuild re-embeds from markdown. Markdown is the source of truth throughout and is never touched.

It will go out in the next release.

Thanks for the report that was embedded in this PR — it had the diagnosis right before the failure was filed.

[Windsander]

Got it, Thanks~