serve engine deadlocks in SleepConditionVariableSRW during a long streaming reply; stall watchdog never fires

Environment

Engine: 0.1.34 ready-made release build (BUILD.json: version 0.1.34, source "release", portable, cuda 13.0, archs 75/86/89/120), engine\strata.exe timestamp 2026-10-02 00:55 UTC. Observed twice live on this exact build. The 0.1.35 changelog doesn't list anything matching this, so I assume it is still present (not yet re-tested on 0.1.35).
Hardware: NVIDIA RTX 5060 Ti 16 GB (sm_120) — the only CUDA device; Intel i5-14600K (6 P-cores + 8 E-cores, no AVX-512); 64 GB RAM; Windows 11 Pro build 26200.
Serve: serve/server.py, OpenAI-compatible endpoint on 127.0.0.1:8080.
Engine args (one line for easy re-use):

--pack <packs\iq3_xxs> --native <shard1> --ple-gguf <shard2> --expert-profile <custom profile> --expert-cache auto --prefill auto --spec 4 --mtp <mtp\rt> --max-context 205824 --kv int8 --kv-resident 32768 --vision --vram-reserve-mib 700 --pcie-frac 0.35 --spec-min-p 0.70 --pool-workers 9 --pool-affinity auto

(--pool-affinity auto was added between the two live observations; the first deadlock predates it, so affinity is not implicated.)

Client: an agentic coding client over OpenAI-compatible SSE: tool-call loops, 34K–85K-token prompts with heavy prefix reuse, reasoning_effort: high → multi-minute <think> streams, occasional client-side cancels.

What happens

Mid-generation, the engine deadlocks: output stops, the GPU drops to 0% while VRAM stays allocated (15.1 GB held), the engine process's total CPU time freezes (every thread blocked), and the request never completes nor errors. The client hangs until the whole service is restarted. The engine log's last line is always a normally-completed earlier request — nothing is written at the moment of the hang, on stderr or stdout.

Happened 4+ times over two days, only with this agentic client; browser chat and short requests never trigger it. Both live-observed cases froze ~74 s into a <think> stream (once at thinking token 2459, once at 9717+).

Evidence (second occurrence, captured live)

Engine process 12+ minutes into the freeze: all 30 threads in Wait (2× EventPairLow, rest Unknown/UserRequest), total CPU time identical across samples.
Full memory dump taken (comsvcs MiniDump /full), parsed without symbols. Main thread:

ntdll.dll!NtWaitForAlertByThreadId+0x14            <- RIP
ntdll.dll!RtlSleepConditionVariableSRW+0x1de
KERNELBASE.dll!SleepConditionVariableSRW+0x38
strata.exe+0x35f118                                <- caller of SleepConditionVariableSRW

The dump was deleted after analysis (64 GB), but I can reproduce, re-dump and pull all 30 thread stacks with a working minidump parser if that helps identify the lock partner.
3. The stall watchdog never fires. After 12+ minutes of a fully frozen process there was no no progress for ... s ... (issue #29) line and no strata-stall-<pid>.dmp. Whatever the main thread waits on, the watchdog's progress accounting does not cover it.
4. serve/server.py wedges with the engine. After I force-killed the deadlocked engine process, the server did not respawn it on the next request (the request handler stays stuck); only restarting the whole server recovers.

Ruled out

RAM/VRAM exhaustion: memtest86 clean; the wedge reproduced with ~24 GB RAM free; engine VRAM was at its planned steady state (520 MiB free, no LOW warning) — and the deadlock is a permanent wait, not a slowdown.
GPU driver TDR: no Event 4101 / LiveKernelEvent at the freeze times.
The client's cancels as the direct trigger of the freeze — a cancel was in flight around the first incident, but the second freeze happened with no cancel; cancelled requests complete cleanly otherwise (v0.1.34's fast cancel works).

Frequency / repro shape

Every occurrence: long agentic session (tool calls + big reused prefixes) with reasoning_effort: high → a thinking stream several minutes long → engine freezes mid-stream. Never in ~50 short chat/browser requests. I cannot reproduce it on demand yet, but it recurs within a day of agentic use.

Asks

Resolve strata.exe+0x35f118 in the 0.1.34 release build (a .pdb/map for release builds — or a hint which std::condition_variable::wait in the serve path sits there — would let me self-resolve and give you the function).
Why the stall watchdog (STRATA_WATCHDOG_S) doesn't cover this wait; ideally a stuck engine should still self-report and dump.
serve/server.py: detect a dead (or force-killed) engine mid-request and respawn, instead of wedging the handler.

Workaround I run meanwhile

An external scheduled task tails the server's stdout (Tee-Object to a file) and restarts the whole service when a [strata] thinking: progress line goes stale for >90 s. Turns the hang into a ~2 min hiccup.

Happy to attach: per-thread stack dump of all 30 threads on the next occurrence, STRATA_TRACE=1 log through a freeze, or the engine + server logs around either incident.

me too

Thanks for the stack. We resolved +0x35f118 against our copy of the 0.1.34 release: it is the static STL's _Cnd_wait, an untimed condition-variable wait on the engine's main thread. The best fit is the engine waiting for its next command between requests while the server waits for that request's end, so the two have lost step. The stall watchdog only counts while a request is running, which is why it never fired. The next release makes the server notice it: no engine output for a while during a request, or a stop that is never acknowledged, now restarts the engine instead of waiting forever.

To find how they lose step, could you, on the next freeze:

post the next 2-3 return addresses above +0x35f118 (we resolve them against the same build);
run with "STRATA_TRACE": "1" in the config's env and post the end of strata-<model>.log plus the server window's last lines;
say whether the request was cancelled or hit a thinking budget just before.

Keeping this open.

serve: end an engine that lost step with the server instead of waitin…

Engine 0.1.37: a silent engine is restarted (#481), Windows AMD count…

0.1.37 adds the server-side safety net: a request whose engine stays silent (default 300 s, longer while a long prompt is read; "engine_silence_s" in the config) or never acknowledges a stop now ends with an error, and the next request restarts the engine. We still want to find how the two lose step: the trace asked for above would help a lot.

@Niko1221

@Niko1221 Deployed v0.1.38 here (release build) — engine_silence_s at its default, and "STRATA_TRACE": "1" is set in the config's env, so the next freeze comes with the trace tail. No freeze has recurred since 0.1.38 went in.

Answers to (3), for the two freezes we captured on engine 0.1.34:

Freeze 2 (2026-10-02 ~07:30 UTC, captured live): the client was still waiting — no cancel was in flight. Thinking was at 2459 of a 131072 budget, nowhere near it. No DONE/ERR was ever logged for that request.
Freeze 1 (2026-10-02 ~16:28 UTC): the client logged stream.failed, status=cancelled about 2 minutes after the freeze began, so the cancel looks like a consequence rather than the trigger. The thinking budget was not hit either.
One more data point that may or may not be related: both freezes happened ~74 s into long streams with reasoning_effort: high.

On the next freeze you will get, as asked:

the ordered strata.exe return addresses above +0x35f118 (my first dump was parsed with a stack walker that collapsed same-module frames, which is why only four frames were posted — the parser now emits every frame in order, and all 30 threads' RIPs with them);
the end of strata-iq3_xxs.log with the trace lines, plus the server console's last lines.
Also noted while testing, in case it is related: terminating a healthy running engine process (45 GiB of the arena locked at 4 KiB pages) has hung in an uninterruptible state for 15+ minutes — TerminateProcess sent, CPU frozen, the locked memory and VRAM held until reboot. May be driver-side and unrelated to the desync; mentioning it since both end in the same "everything waits forever" symptom.

hip: hipBLASLt tuning table for gfx1100 (RX 7900 XTX) with ROCM 10.2.…

serve: end an engine that lost step with the server instead of waitin…

Engine 0.1.37: a silent engine is restarted (#481), Windows AMD count…

hip: hipBLASLt tuning table for gfx1100 (RX 7900 XTX) with ROCM 10.2.…

serve: end an engine that lost step with the server instead of waitin…

Engine 0.1.37: a silent engine is restarted (Niko1221#481), Windows A…

hip: hipBLASLt tuning table for gfx1100 (RX 7900 XTX) with ROCM 10.2.…

serve: end an engine that lost step with the server instead of waitin…

Engine 0.1.37: a silent engine is restarted (Niko1221#481), Windows A…

serve: time out the untimed reads of the engine's READY line and the …

hip: hipBLASLt tuning table for gfx1100 (RX 7900 XTX) with ROCM 10.2.…

serve: time out the untimed reads of the engine's READY line and the …

@Niko1221

Seems like the env STRATA_ENGINE_READY_S=0 only affects the engine's start.
On my potato gamer, the env allows the engine to start after some time, but a request that needs to read the entire conversation to compact gives me the message:

[...]
[strata] reading the prompt: 14,223 of 49,581 tokens, 854 s so far
[strata] the engine said nothing for 1871 s during the request; the server ended the engine. The engine and the server lost step (issue #481; a very slow PC can raise "engine_silence_s" in the config, 0 = wait forever). If it happens again, please add the end of the engine log to github.com//issues/481. The next request starts the engine again. Its log: D:\Downloads\models\Strata-main\strata-coder-iq1_m.log
[strata] done: 0 tokens in 2738 s (0.0 tok/s) (error, cancel=False)

Yes, the engine takes an awful amount of time to start and the prompt reading is very poor here.
But the decoding is 'light years away' from any other options I have. So, I prefer this.

So, the env should avoid the server ending the engine on any request. I set it up asking for that.

Thank you for your project @Niko1221 !

serve: the solo generate path reads the engine process once too

serve: a request whose engine process was taken says the engine is gone

Adding matching evidence from the HIP / gfx1201 path (Radeon AI PRO R9700, IQ3_S, engine 0.1.39) for the payload the issue asks for.

The server line:

[strata] the engine said nothing for 300 s during the request; the server ended the engine. ...
  (issue #481; a very slow PC can raise "engine_silence_s" in the config, 0 = wait forever)

The engine log at that moment: it had stopped making progress during a long prefill, and the watchdog's stall report shows the host parked on a GPU operation — not on disk, not on the expert pool:

strata serve: no progress for 300 s during a request (reading the prompt (batched):
  waiting for the GPU (attention, router) at layer 37 of the prompt chunk from token 5101)
  - stopping the engine so the server starts it again (issue #29)
strata serve: stall report (engine 0.1.39): stage "reading the prompt (batched): waiting for the GPU
  (attention, router) at layer 37 of the prompt chunk from token 5101" for 300 s;
  0 layers served since the last finished step (0 = stopped, more = slow)
  expert pool: epoch 0, batch epoch 0: 0 of 0 jobs claimed, 0 done; 11 of 11 workers parked, 11 sleeping; mode 0
  verify window (last window, not the current stage): 0 tokens at position 0, host at layer step 1;
    the GPU rang 0; flags: served 0, plan (A) 0, copies (B) 0
  threads waiting on the disk (state D): 0 of 67
  memory: 29297 MiB resident, 13 MiB in swap, 28000 MiB RAM available; 266165 major page faults so far
  2 s later:
  expert pool: 0 of 0 jobs claimed, 0 done; 11 of 11 workers parked, 11 sleeping; mode 0
  verify window: 0 tokens ... the GPU rang 0 ... host idle for 510650 ms

Notes that may help narrow it: every expert worker parked and asleep, verify window "GPU rang 0", 0 threads in D-state, and low swap. That reads as the host blocked in a GPU sync (cudaStreamSynchronize in the prefill path) rather than a scheduler/pool problem. Config: engine 0.1.39, gfx1201, 131072 ctx, --kv int8 --kv-resident 32768, --vram-reserve-mib 3072. It happened once at 3072 MiB reserve (many more at lower reserve; 700 MiB also drove a ~7x driver-paging slowdown). Happy to attach the full engine-log tail if useful.

Thanks for the report. To keep the tracker current we are closing issues older than three days: 0.1.44 is out with a long list of fixes, and many older reports are covered by it or by 0.1.43.1.

If this still happens on 0.1.44, please leave a comment here with your engine log (and the version line it prints), and we will reopen it. A new issue that links this one works too.

Navigation Menu

Repository navigation

serve engine deadlocks in SleepConditionVariableSRW during a long streaming reply; stall watchdog never fires #481

Environment

What happens

Evidence (second occurrence, captured live)

Ruled out

Frequency / repro shape

Asks

Workaround I run meanwhile

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

serve engine deadlocks in SleepConditionVariableSRW during a long streaming reply; stall watchdog never fires #481

Description

Environment

What happens

Evidence (second occurrence, captured live)

Ruled out

Frequency / repro shape

Asks

Workaround I run meanwhile

Activity

wxaef commented on Oct 2, 2026

Niko1221 commented on Oct 2, 2026

Niko1221 commented on Oct 2, 2026

gu-feng commented on Oct 3, 2026

Yoshimit commented on Oct 8, 2026

rebuilt commented on Oct 11, 2026

Niko1221 commented on Oct 13, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions