Skip to content

Engine death leaves the server's max_context at 0, and the next request gets a misleading "leaves no room to answer in the context (0)" 400 #344

Closed

Description

@backstable

Summary
When the engine dies, the server keeps engine.max_context == 0 and rejects the next request with
prompt (41732 tokens) leaves no room to answer in the context (0); requests are never truncated
That message points at the prompt, but the prompt has nothing to do with it. The real condition is "the engine is not up". With --max-context 131072 and a 41,732-token prompt there is ~89k tokens of room; the number in parentheses is the bug.
Clients surface this to users as "custom model error, switch model or retry", so the natural reaction is to switch models or shrink the prompt — both wrong. The correct action is to wait for the engine to come back.
Where it comes from
serve/server.py:

169 (StrataEngine.init)

self.max_context = 0
...

185 only place max_context is ever assigned: the engine's READY line on stdout

if line.startswith("READY"):
f = line.split()
self.max_context = int(f[1])
...

189

if self.max_context <= 0:
raise RuntimeError("the engine exited before it was ready ...")
So max_context is set once, at startup. When the engine later exits, the server logs it and restarts it on the next request — but nothing resets or preserves max_context, and the field is still 0 while the restart is in flight:

752

room = self.engine.max_context - CTX_SLACK - len(ids) # CTX_SLACK = 8 (line 53)
if max_new is None or max_new <= 0 or (self.fit_max_tokens and room < 1):
if room < 1:
raise ValueError(f"prompt ({len(ids)} tokens) leaves no room to answer in the context "
f"({self.engine.max_context}); requests are never truncated")
room becomes 0 - 8 - 41732. max_new <= 0 and room < 1 are both true, so it takes the room < 1 branch and reports the prompt length as the cause. Note this also defeats fit_max_tokens: true, whose stated purpose is "a shorter completion beats a 400": the room < 1 check runs before any fitting can happen.
What we saw
Timeline from journalctl -u strata-iq3xxs and the engine log (engine 0.1.27 at the time):
12:03:4x [strata] the engine reported an error: cancelled
strata serve: prompt 66873 tokens = 0 reused + 66873 read ... 0 generated ... (cancelled)
strata serve: cancelled
12:03:55 [strata] the engine stopped unexpectedly (exit code 1). ... The next request starts the engine again.
12:04:38 <-- user sends a message: gets the 400 above
12:04:39 [strata] the engine had stopped (exit code 1); starting it again (a minute or two) ...
12:05:1x [strata] ready: http://127.0.0.1:18080/v1 (..., context 131072 tokens)
12:05:15 same user message succeeds
The user hit a 44-second window between "the engine stopped" and "the engine is ready again".
The engine death itself was the 0.1.27 cancel bug (fixed by 0.1.28 — see the note in setup.py's version comment). On 0.1.30 we could not reproduce it, but the misleading message is independent of that: any engine death, from any cause (OOM, driver reset, manual kill) produces the same 400.
Reproduction
Any way of killing the engine works; the simplest:
sudo systemctl restart strata-iq3xxs # or: kill the engine PID

while it reloads, send a normal-sized request to the server

curl -sS -X POST http://127.0.0.1:18080/v1/chat/completions
-H 'Content-Type: application/json'
-d '{"model":"...","messages":[{"role":"user","content":"hi"}],"max_tokens":16}'
→ {"code":-32603, ..., "message":"prompt (N tokens) leaves no room to answer in the context (0); ...","statusCode":400}
Suggested fix
Distinguish "no engine" from "not enough context". Either is small:

  1. Report the real reason. In the room < 1 branch, check self.engine.max_context <= 0 first and raise a distinct error, e.g. 503 engine is not up (starting: a minute or two). The server already has the wording for this in its own startup messages.
  2. Or keep the last known good max_context across restarts, so the arithmetic stays sane and the failure surfaces where it belongs (as a connection error, not a context error).
    Option 1 is the smaller change and fixes the user-visible misdirection, which is the actual harm here.
    Happy to send a PR for either if you want it.

Activity

  1. Niko1221 commented on Oct 1, 2026

    @Niko1221
    Owner

    Confirmed: restart() marks the engine alive before it is READY. 0.1.32 marks it alive only after READY, and a request during a restart gets a 503 "engine is starting".

  2. Niko1221 commented on Oct 1, 2026

    @Niko1221
    Owner

    Fixed in 0.1.32: a request during a restart gets a 503 "the engine is starting". Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions