Self-hosted server: memory agent permanently broken when a boot loses the LLM-provider init race; error swallowed from local logs; retry cron terminal-fails recoverable docs #1208

Environment

• supermemory-server v0.0.3, linux-x64, installed via curl -fsSL https://supermemory.ai/install | bash
• systemd-managed, self-hosted mode, embedded pglite store, local Xenova embeddings, one external LLM key configured (verified valid, pay-as-you-go, no rate limiting observed)

Bug 1 — startup race leaves the memory agent permanently broken for the process lifetime

If the HTTP server starts serving (and the ingest queue starts draining) before the LLM
provider registry is initialized, every memory-agent call in that process fails in 1–8 ms
(synchronous, pre-network) and the process never recovers until restarted.

Evidence from journalctl across three boots of the same config/key/docs:

• Boot A: failures began 1 second after listening; 498 docs failed in 90 s, all memory agent failed (1-8ms); zero completions for the process lifetime.
• Boot B (restart, no changes): 2,200+ completions, zero failures.
• The losing boot logs WARNING: No LLM providers configured! ... The server will start but ALL AI features will fail. at startup — but only warns; ingest is not gated on provider readiness.

Suggested fix: gate workflow/ingest start on provider-registry readiness, and fail fast
(exit non-zero) if the registry is empty when a job executes — under Restart=always a
crash-and-revive beats a process that limps forever with all AI features dead.

Bug 2 — the caught error never reaches local logs

The per-document failure handler captures the real error into analytics/telemetry spans,
but the only lines that reach stdout/journald are:

[Workflow] Document <id> memory agent failed (2ms)
WARN [Workflow] Self-hosted memory agent failed for document, skipping memory generation

Neither includes the error message or stack ([llm] No LLM providers available ... in our
case). Self-hosted deployments have no analytics sink, so the cause is invisible — this
cost us days of misdiagnosis (we suspected LLM quota). One-liner: include
err.stack ?? err.message in the local log line.

Bug 3 — retry cron terminal-fails docs that are fully recoverable from the store

Docs whose processingMetadata lacks a retry-params key (pre-existing docs, or params blob
missing on disk) are marked:

[Cron] Marked <id> as failed: Permanently failed (no retry params key)
[Cron] Marked <id> as terminally failed: No retry params on disk (pre-existing doc or store cleared)
[Cron] Processing retry done: 0 retried, 500 failed, 0 alive

…every 30 minutes, forever — even though content, containerTags, and customId are all
still in the store and re-ingesting the same content via POST /v3/documents (customId
upsert) processes fine. Suggested fix: when retry params are missing but the stored document
has content, rebuild the ingest job from the stored row (the same path the public POST uses)
instead of marking it permanently failed.

Impact seen

~3,400 documents stuck failed on a healthy store; recovered externally by re-posting every
failed doc through POST /v3/documents with its original customId/tags.