ai_chat_send_to_idle_test times out with assistant_count=0 — blocks the pre-push gate (4/5 attempts today)

Summary

packages/desktop-app/src-tauri/tests/ai_chat_send_to_idle_test.rs's only test, ai_chat_send_reaches_idle_with_no_stuck_processing_state (ADR-048 Tauri-seam suite, part of the mandatory scripts/test-gate.ts pre-push gate), is currently timing out at its 300s deadline with assistant_count=0 in 4 of 5 consecutive git push attempts today, blocking every push from this checkout regardless of what changed.

What was observed

Working an unrelated packages/agent change (core#2190 — a comment-only edit to tools.rs plus a new golden-case trio under goldens/ablation/, no touch to desktop-app, ai-chat, or the daemon's event-watcher/inference-dispatch code at all), the pre-push gate failed 5 times in a row:

FAILED — ai_chat_send_reaches_idle_with_no_stuck_processing_state, 300s timeout, last status="idle", assistant_count=0
FAILED — different test (model-store.test.ts, an unrelated frontend fake-timer flake) — not part of this report
FAILED — same test, same symptom, finished in 302.86s
FAILED — same test, same symptom, finished in 303.83s
FAILED — same test, same symptom, finished in 303.14s

Panic message (identical each time):

thread 'ai_chat_send_reaches_idle_with_no_stuck_processing_state' panicked at packages/desktop-app/src-tauri/tests/ai_chat_send_to_idle_test.rs:98:13:
ai-chat turn did not reach idle-with-new-reply within 300s; last status="idle", assistant_count=0

assistant_count=0 means the polled node never gained a new assistant message at all — not that generation was slow, but that the turn appears to never have produced (or never dispatched) a reply within the daemon under test.

Why this doesn't look like ordinary machine-load flakiness

The test's own doc comment already documents a prior class of stall (ministral-3b-q4km "reproducibly stalled to 0 tokens generated within the 300s timeout, independent of machine load") that led to a model swap, so this suite has known history here — but the current lock (per ADR-056) is gemma-4-E4B-it-Q4_K_M.
In the same session, the same model was driven successfully 27/27 times via nodespace_agent::golden::runner's direct LlamaChatInferenceEngine path (packages/agent/src/bin/golden_runner.rs) for core#2190's own measurement — so the model itself generates correctly through that path. This test instead drives a full nodespaced daemon through the real Tauri command layer and polls get_node, so the failure looks like it's somewhere in the daemon's event-watcher / inference-turn-dispatch path (or a resource contention specific to running a second full daemon+model instance), not in the model or in llama.cpp itself.
4 of 5 attempts hit the exact same symptom and near-identical duration (~303s), which reads as systematic rather than one-off.

What this is not

Not caused by core#2190's own diff — verified as a pure comment addition to tools.rs (zero functional change) plus new golden-case fixtures. The pre-push gate reproduces this failure regardless of whether that diff is present.

Ask

Someone who owns the ADR-048 Tauri-seam suite / daemon ai-chat dispatch path should reproduce directly (cargo test -p nodespace-app --test ai_chat_send_to_idle_test) outside a loaded pre-push-gate run, and check whether this is transient contention in a specific environment or a live regression in turn dispatch. Until then, this blocks the mandatory pre-push gate for any change touching this checkout.

Root cause: the failing runs spawned a 2026-07-24 `nodespaced`, not a build of `main`

Reproduced the reported signature verbatim and traced it. This is not a regression in the daemon's turn-completion path — current main's daemon passes this test repeatedly, and a month-old daemon binary fails it in exactly the way reported.

1. Current `main` passes

Fresh worktree at 5c4f3cba, daemon from cargo build --bin nodespaced:

run	result
isolated `cargo test -p nodespace-app --test ai_chat_send_to_idle_test`	ok, 39.65s
the gate's own step 5: `cargo test -p nodespace-app -- --test-threads=1` with `NODESPACED_TEST_BIN`	ok, 30.79s — whole suite green
same again after the fix below	ok, 22.88s — whole suite green

2. The binary the failing runs used is from 24 July

26,972,864  24 Jul 11:43  core/target/debug/nodespaced                               md5 379741d5…
26,972,864  24 Jul 11:43  core/…/src-tauri/binaries/nodespaced-aarch64-apple-darwin  md5 379741d5…
26,972,864  21 Aug 13:34  core-worktrees/issue-2190/target/debug/nodespaced          md5 379741d5…
98,952,648  24 Aug 08:51  fresh worktree/target/debug/nodespaced                     md5 93706f64…

The first three are byte-identical — copied around, never rebuilt. A real debug nodespaced today is ~99 MB, not 27 MB.

nodespace-app-test-support::resolve_daemon_binary() resolved NODESPACED_TEST_BIN, else the hand-staged src-tauri/binaries/ sidecar — so the isolated repro in the primary checkout (no env override) spawned the 24 July sidecar. And whatever cargo build --bin nodespaced did at gate step 2, the binary the gate then pointed NODESPACED_TEST_BIN at was byte-identical to that same July build; cargo's freshness check is mtime-based, so build outputs copied into a fresh worktree date newer than its sources and register as up to date.

3. Pointing the test at that July binary reproduces the report exactly

ai-chat turn did not reach idle-with-new-reply within 300s; last status="idle", assistant_count=0;
node={… "messages":[{"content":"Reply with exactly one word: OK","role":"user"}], "status":"idle", "version":3}
test result: FAILED. finished in 309.56s

and the daemon's own log, one second into the turn:

…ded: Prompt uses 6032 tokens but context window is 4096
INFO nodespace_daemon::…::local_agent_service: ai-chat turn complete

The turn failed with ContextOverflow and that daemon reset status to idle without appending anything — the pre-#2168 behaviour. #2168 (39d4f08a, merged 2026-08-19) is the change that made a failed inference turn append a visible assistant message instead of going silently idle; its regression test failed_inference_turn_surfaces_a_visible_error_not_silent_idle pins the identical error string. Any daemon built before 2026-08-19 has the old behaviour, and status="idle", assistant_count=0 is its signature. version: 3 in the node dump above (create → user message → status reset) confirms the silent status write.

Ruled out

The 08-21→08-23 packages/agent merges — fresh main passes; reproducing the failure needs the July binary.
agent_loop.rs completion paths — on main every run_turn return carries non-empty text (including the no-usable-final-text fallback), so the Ok arm always appends and the Err arm appends the failure message. The only remaining zero-append idle paths are cancellation (deliberate) and a failed append_assistant_message; neither is reachable here.
A wire-shape mismatch dropping messages — nodespace_core::models::AiChatMessage and nodespace_types::ai_chat::AiChatMessage are field-for-field identical, and the node dump confirms the node genuinely held no assistant message.
Machine load — the July binary fails on an idle machine; the current one passes under the full serialized suite.

Fix

#2258 — no production code changed. The ADR-048 fixture now prefers target/debug/nodespaced over the sidecar and refuses to spawn a daemon older than the sources compiled into it, failing in 0.01s with both filenames and how to rebuild, rather than surfacing minutes later as a behavioural failure blamed on whatever is being pushed. Plus two diagnosability fixes that made this traceable at all: RUST_LOG is forwarded to the spawned daemon, and the ai-chat timeout panic now dumps the polled node instead of two derived counters.

If you hit this on this machine

core/target/debug/nodespaced and core/packages/desktop-app/src-tauri/binaries/nodespaced-* are still the July binaries — rm them and rebuild before running the ADR-048 suite. With this PR merged you get told that instead of having to work it out.

Navigation Menu

Repository navigation

ai_chat_send_to_idle_test times out with assistant_count=0 — blocks the pre-push gate (4/5 attempts today) #2224

Summary

What was observed

Why this doesn't look like ordinary machine-load flakiness

What this is not

Ask

Root cause: the failing runs spawned a 2026-07-24 `nodespaced`, not a build of `main`

1. Current `main` passes

2. The binary the failing runs used is from 24 July

3. Pointing the test at that July binary reproduces the report exactly

Ruled out

Fix

If you hit this on this machine

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

ai_chat_send_to_idle_test times out with assistant_count=0 — blocks the pre-push gate (4/5 attempts today) #2224

Description

Summary

What was observed

Why this doesn't look like ordinary machine-load flakiness

What this is not

Ask

Activity

mstomar125 commented on Aug 24, 2026

Root cause: the failing runs spawned a 2026-07-24 nodespaced, not a build of main

1. Current main passes

2. The binary the failing runs used is from 24 July

3. Pointing the test at that July binary reproduces the report exactly

Ruled out

Fix

If you hit this on this machine

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Root cause: the failing runs spawned a 2026-07-24 `nodespaced`, not a build of `main`

1. Current `main` passes