Repository navigation
写入失败被上报为成功:cascade worker 不可恢复时 flush 仍返回 extracted;且失败无重试上限 #364
Description
一句话
cascade worker 处于持续失败状态时,写入 API 仍然返回成功 —— POST /api/v1/memory/add 返回 200、POST /api/v1/memory/flush 返回 {"status": "extracted"},而实际没有任何内容落库。宿主应用无法从 API 契约上察觉记忆已经停止工作。同时,失败条目没有重试上限,单次运行可产生数百条完全相同的不可恢复日志。
这是从 #337 排查中分离出来的两个独立问题。#337 的具体成因(subject_vector schema 漂移)已由 #354 检测 + 恢复,但下面两点不随之解决。
1. 写入失败被上报为成功
复现
在一个 episode 写入必然失败的库上(例如 #337 的 schema 漂移状态):
curl -X POST localhost:1995/api/v1/memory/add -H 'Content-Type: application/json' \
-d '{"user_id":"default","app_id":"default","session_id":"probe",
"messages":[{"role":"user","sender_id":"default","timestamp":1769500000000,"content":"probe"},
{"role":"assistant","sender_id":"agent","timestamp":1769500000001,"content":"ack"}]}'
# -> 200 {"data":{"message_count":2,"status":"accumulated"}}
curl -X POST localhost:1995/api/v1/memory/flush -H 'Content-Type: application/json' \
-d '{"user_id":"default","session_id":"probe"}'
# -> 200 {"data":{"status":"extracted"}}同时服务端日志里是:
cascade_worker_unrecoverable kind=episode ...
RuntimeError: lance error: LanceError(IO): Execution error: Spill has sent an error
随后检索写入的内容,搜不到;episode 表行数不变。
影响
"status": "extracted" 在语义上表示抽取已完成,调用方据此认为写入成功。实际情况是这条记忆永久不会进入索引。
在我们的场景里,这导致 episode 写入静默失败了 18 天(表停在 7-09,期间零写入)才被发现 —— 因为:
- 写入侧 API 一直返回成功;
- recall 仍能召回旧数据(其他 kind 如
atomic_fact/foresight写入正常),看起来"记忆还活着"; - 宿主(Raven)的记忆后端是 fail-open 的:捕获异常后返回空结果,不向上抛,所以它自己的链路追踪里这些节点也全是绿的。
三者叠加,故障完全不可见。
为什么 #354 不覆盖
#354 的处理方式是启动期拒绝启动(检测到 schema 漂移就抛错),对这一类成因有效。但只要写入失败源于其他原因(嵌入服务异常、磁盘、下游超时等),cascade worker 依旧会静默失败,而 API 依旧返回 extracted。
建议
flush的返回值反映真实结果,例如在有条目进入failed/ 不可恢复状态时返回partial/degraded,并带上失败计数;- 或提供一个可查询的健康/降级状态(例如
/health中包含 cascade 队列的 failed 数量),让宿主能够主动探测; - 至少:当队列中存在不可恢复条目时,不要在响应中使用
extracted这种表示成功的措辞。
2. 不可恢复失败没有重试上限
同一批 md 条目在单次运行中产生了 330 条 cascade_worker_unrecoverable kind=episode,内容完全相同。这些重试没有任何成功可能(schema 漂移是确定性失败),只会持续消耗 CPU、写入日志噪声,并淹没其他有用日志。
建议
对同一条目引入重试上限或退避:达到上限后标记为终态 failed(附原因)并停止重排,让 cascade fix / cascade rebuild 去处理,而不是无限循环。
环境
- everos 1.1.3(1.2.0.dev1 表现相同)
- lancedb 0.33.0 / 0.34.0,pyarrow 25.0.0
- Python 3.12.13,macOS 26.3 arm64
- 宿主:Raven(通过
everos-memory插件走 HTTP 接入)
相关
- cascade worker fails to index episodes: LanceDB
Spill has sent an error(IO) #337 —— 具体成因(episodesubject_vectorschema 漂移) - fix(lancedb): detect schema type drift and add cascade rebuild recovery #354 —— schema 漂移检测 +
cascade rebuild恢复
Activity
arkh-node commentedon Aug 1, 2026
This is a good report — the three-layer invisibility you describe (write API returns success, recall still serves old rows, host is fail-open) is exactly what makes this class of failure survive for weeks. I hit the same shape in a different pipeline recently: a job logged
finishedwhile its output directory stayed empty, and it took a separate consistency check to notice.Reading the tree, I'd like to offer a narrower framing of part 1, and pick up the piece that seems both smallest and most useful.
flushisn't lying inside its own layer — the gap is between layers.memorizewrites the markdown and legitimately reportsextractedfor that step; indexing into LanceDB happens later, in the cascade worker. So"status": "extracted"is accurate about what it did, and misleading about what the caller assumes. Wideningflush's return type topartial/degradedwould mean the write path has to await or poll cascade state, which couples two deliberately decoupled stages.The information already exists — it just has no HTTP exit.
md_change_state_repo.queue_summary()already computesfailed_permanent(status='failed' AND retryable=FALSE) alongsidepending,done,failed_retryable, and the CLI surfaces all of it incascade status. A host that talks to EverOS over the CLI can already see the 18-day outage; a host that talks over HTTP — like Raven througheveros-memory— cannot see it at all. That asymmetry looks like the actual defect: the system knows it is degraded and has no way to say so to its main integration surface.So the proposal for part 1, in the spirit of your third suggestion:
GET /healthwith acascadesection carrying the existingQueueSummaryfields (pending,done,failed_retryable,failed_permanent, and the LSN lag the CLI already derives);failed_permanent > 0means "memory writes are landing in markdown but not in the index, and no retry will fix it without operator action."That gives fail-open hosts something to actually check, without making the write path synchronous on the indexing path.
On part 2 (no retry ceiling): the current code already marks unrecoverable rows terminal — the
except Exceptionbranch callsmark_failed(retryable=False), andclaim_pending_batchshouldn't be picking those up again. So 330 identicalcascade_worker_unrecoverablelines for the same entries suggests the rows are being re-enqueued from somewhere else rather than retried in place — most likely the scanner re-observing the same md files by mtime and upserting them back topending. If that's what's happening, the ceiling belongs at enqueue time (don't re-enqueue a path that is terminally failed unless its content hash actually changed), not in the retry loop. I'd rather confirm that from a reproduction than guess in a patch, so I'd keep it separate from the health work.Happy to open a PR for the
/healthpart if that direction sounds right to you — with a test that a permanently-failed row shows up in the endpoint. And if you'd prefer the whole thing to wait until #337/#354 settle, that's fine too; I'd rather ask first than send a patch you have reasons not to want.gloryfromca commentedon Oct 20, 2026
Triage note: the second of the two defects here is already fixed, and the first belongs with a group of issues asking the same question.
The retry cap landed the day after this was filed.
42629df(2026-07-28, #366) added a cross-cycle retry budget,_MAX_TOTAL_RETRIES = 12in the cascade worker. A row that reaches it is markedretryable=Falsewith a singlecascade_worker_retry_budget_exhaustedwarning naming the path, the kind and the count, instead of being re-attempted on every scanner cycle. The "hundreds of identical unrecoverable log lines in one run" this issue reports is bounded by that budget. Nobody came back to say so, hence this note.What remains is the success/failure contract, and it is the same question four open items are asking from different sides:
/flushreturns{"status": "extracted"}while the cascade worker is failing permanently — the caller is told the memory was extracted when it will never be indexed500 INTERNAL_ERROR, so a caller cannot tell "the server is broken" from "try again in a moment"/addrequestThese cannot be fixed one at a time without fighting each other: giving #498 a retryable status code is an instruction to clients to retry, and retry safety is exactly what #274 and #516 say is missing. The honest shape is one decision about what the write API promises — when a 2xx means "durably accepted", what a caller is expected to do on a timeout, and what identifies a request so a replay overwrites instead of appending — and then an implementation across all four.
So: this issue now means only the first defect. No change is needed for the second.