Repository navigation
fix(profile): 通过父引用投影解析聚类成员，批量读取时跳过向量列 - #490
Merged
gloryfromca merged 7 commits intoOct 15, 2026
Merged
gloryfromca merged 7 commits into
gloryfromca merged 7 commits into
Conversation
gloryfromca
self-requested a review
October 15, 2026 07:55
gloryfromca
approved these changes
Oct 15, 2026
_select_via_cluster fetched full episode rows, both embedding vectors included, for every member of every fresh cluster, only to read parent_type and parent_id. Clusters can hold thousands of members, so the read dominated memory for the profile extraction path. find_by_owner_entries gains an optional columns projection on the index contract (raw dicts when set, typed rows otherwise, as list_by_owner_after_ts already does). LanceDB pushes the projection down to the query; Milvus still loads full rows and only reshapes them to the same dict contract. The profile cluster path asks for parent_type and parent_id only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Adding the columns projection widened the public return type to list[Any] (protocol, router) or an unconditional union (LanceDB, Milvus), so callers that omit columns lost list[T]. Overload find_by_owner_entries on every layer: columns omitted or None returns list[T]; a column sequence returns list[dict[str, Any]]. Runtime behaviour is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
patch produced offline; applied mechanically
patch produced offline; applied mechanically
Bulk readers materialised every row's 1024-D vectors as Python floats even though none of them use the vectors. On 2026-10-12 this pushed the cascade daemon past MemoryHigh twice: profile cluster resolution fanned out 18 concurrent full-row reads per memory/add, and a burst of adds drove anon memory from <1G to 7.3G in five minutes, freezing every thread in mem_cgroup_handle_over_high. find_where and find_by_owner_entries gain skip_vectors on every backend (typed rows returned with the vector columns left out): the cascade daily-log diff, agent-skill supporting-case hydration and the reflection episode fetch now read lean rows. None of these paths write the rows back. Together with the parent-ref columns projection this keeps vector payloads out of every hot bulk-read path.
Three gaps in the projection, all on paths the change exists to fix. Milvus find_by_owner_entries forwarded skip_vectors but defaulted it to False, and its only caller passes columns without it -- so a profile cluster resolution loaded every 1024-D vector and discarded it on the next line. A columns list that names no vector column cannot need the payload, and Milvus charges for it on the fetch, so decide it there. The daily-log handler's skip_vectors=True had no coverage: dropping it left all 258 cascade tests green. The fake already took the kwarg; record it and assert the handler passes it. The Protocol's implementation signature omitted skip_vectors while both its overloads declare it, and the router forwards it unconditionally -- a third backend written against that signature would raise TypeError. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The projection landed twice. LanceIndexContract overrides find_by_owner_entries and every LanceDB repo declares it first, so the chassis copy was unreachable -- and it was the deadline-covered one, leaving the live copy outside _READ_TIMEOUT_SECONDS on the profile cluster hot path. Drop the dead branch, give the chassis a project_rows helper that keeps the deadline, and have the contract call it. Two tests the projection was missing. The backend-parity test for columns was pinned to lancedb; unpinned, it covers the Milvus dict shape whenever EVEROS_TEST_MILVUS_URI is set. The Milvus columns-to- skip_vectors inference had no coverage at all. And one for the hazard the whole change creates: a row read with skip_vectors has vectors of None because they were not read, not because the record has none, so upserting one back destroys a stored embedding with no error and no log. Nothing does this today; this pins it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
gloryfromca
force-pushed
the
feature/profile-parent-ref-projection-v2
branch
from
October 15, 2026 09:08
bdca0ec to
1d2cd8f
Compare
gloryfromca
added a commit
that referenced
this pull request
Oct 21, 2026
#492 (the cascade handler limit held per drain pass, not per worker) and #490 (resolving a cluster's members read every member's vectors) merged after 1.4.2 without a changelog entry. Both go under Fixed, and the lead paragraph names the second. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
gloryfromca
added a commit
that referenced
this pull request
Oct 21, 2026
* chore(release): v1.5.0 Moves the Unreleased entries under `## [1.5.0] - 2026-10-21` with the release page's lead paragraph, and adds one note to Upgrade: the first start creates the `memorize_inbox` table, and `async_mode` is opt-in. Version bump in `pyproject.toml`, `docs/openapi.json` (regenerated after `uv sync`) and the project's line in `uv.lock`. 1.5.0rc1 (same code) went through full release acceptance on macOS, Windows and Linux, a PyPI-resolved install smoke, and an upgrade check on a root built by 1.4.2. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(changelog): add the two 1.5.0 fixes the notes missed #492 (the cascade handler limit held per drain pass, not per worker) and #490 (resolving a cluster's members read every member's vectors) merged after 1.4.2 without a changelog entry. Both go under Fixed, and the lead paragraph names the second. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: zhanghui <23442919+gloryfromca@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
描述：
背景
画像提取在把聚类成员解析到父级记忆时，原有路径会加载完整的 episode 行（包括两组 embedding 向量），实际只需要读取 parent_type 和 parent_id。聚类规模较大时，会造成不必要的 I/O 和内存占用。
改动
• 在索引仓库接口中新增可选的 columns 投影参数；不传 columns 时仍返回原有的类型化行（保留 typed-row 重载）。
• 在 LanceDB 中通过 .select(columns) 在物化之前下推投影。
• 画像聚类解析路径只请求 parent_type 和 parent_id。
• 为批量行读取新增 skip_vectors，维护类扫描可以不再物化两组向量。
• 保持路由层行为一致，并补充接口契约、策略、双触发、cascade 和集成测试。
• Milvus 兼容投影后的返回结构，路由调用不会因此报错。
验证
• 定向回归（cascade / infra index / strategies / reflection / OME 集成）：基于 upstream main @ a345d08，943 passed
• Ruff check：通过
• Ruff format check：通过
已知限制
存储层优化仅对 LanceDB 生效。Milvus 接受 columns 参数只是为了兼容返回结构，内部仍会物化完整的类型化行，再在 Python 中裁剪；本 PR 不声称为 Milvus 实现了存储层投影或带来性能提升。
范围
本 PR 只包含通用的仓库层、画像和批量读取改动及对应测试，不包含本地补丁记录、环境文件、部署配置及 OpenClaw 专用内