Stalled model-provider streams abort agent rounds at the 30-minute guard with no automatic retry

What is failing

Model-provider streaming responses stall mid-stream (the endpoint stops emitting tokens and never completes the SSE sequence), so StreamStallGuardChatClient fires its configured 30-minute maximum-round-duration abort and kills the agent thread-execution round that was waiting on the stream. All observed rounds were the review-pull-request-systemorph-meshweave-* triage thread on the memex portal. The guard itself behaves correctly — nothing in the stack indicates a defect in our wrapping clients; the underlying failure is the provider stream going silent, and recovery today is manual ("resubmit to retry").

Probable cause

Provider-endpoint stall during long streaming rounds (high confidence). The abort is the guard's designed behaviour (00:30:00 max round duration, round #147 in all three samples), and the inner TaskCanceledException is the guard's own cancellation token firing while the provider stream produced nothing further. StreamStallGuardChatClient.GetStreamingResponseAsync and ProviderCallTimingChatClient are pass-through decorators around the failing stream. A contributing question: three stall-aborts clustered on the same thread kind within ~70 minutes may indicate a provider outage or a systematically over-long prompt for the PR-review thread, but the sample count (3) does not let me distinguish those — an engineer should check provider metrics for the window.

Impact

Low. Three aborted rounds on a single pod over ~70 minutes, all one agent thread family (automated PR review). Each abort costs up to 30 minutes of wasted round time plus a lost thread round that must be manually resubmitted; no data loss or user-visible portal breakage.

Where to look

plugins-repo/src/MeshWeaver.AI/StreamStallGuardChatClient.cs — timeout/abort path (~lines 102, 188); candidate improvement: automatic in-round retry of the provider call before aborting
plugins-repo/src/MeshWeaver.AI/ProviderCallTimingChatClient.cs — provider timing data to confirm the stall window
Provider-side metrics for the affected rounds to confirm the endpoint stall

Possibly related earlier incidents with the same log shape: Admin/_LogIncident/627b37f452758c93, Admin/_LogIncident/652e6089f34dfbd0, Admin/_LogIncident/801f035b572d2f65 — if those are the same defect, this ticket can be closed as a duplicate once confirmed.

Evidence


Fingerprint	`8bfa7322a2e2815c`
Category	`MeshWeaver.AI.AgentChatClient`
Severity	Error
Exception	`System.TimeoutException`
Top frame	`MeshWeaver.AI.StreamStallGuardChatClient.GetStreamingResponseAsync(IEnumerable`1 messages, ChatOptions options, CancellationToken cancellationToken)+MoveNext()`
Namespace	`memex`
Pods	`memex-portal-deployment-5c476fdbf8-6hr6s`
Occurrences	3
First seen	2026-09-28 11:14:24Z
Last seen	2026-09-28 12:24:28Z
Routing	not determined — no configured route matches the category `MeshWeaver.AI.AgentChatClient`. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject.

Recent log lines

2026-09-28 11:14:24Z memex-portal-deployment-5c476fdbf8-6hr6s fail: MeshWeaver.AI.AgentChatClient[0]
      [ThreadExec] ERROR: 11:14:24.604 threadPath=Hosting/Triage/_Thread/review-pull-request-systemorph-meshweave-f444
      System.TimeoutException: AI streaming exceeded the maximum round duration of 00:30:00 and was aborted (#147). The model endpoint stopped responding mid-stream; resubmit to retry.
       ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
         at MeshWeaver.AI.StreamStallGuardChatClient.GetStreamingResponseAsync(IEnumerable`1 messages, ChatOptions options, CancellationToken cancellationToken)+MoveNext() in /home/runner/work/MeshWeaver/MeshWeaver/plugins-repo/src/MeshWeaver.AI/StreamStallGuardChatClient.cs:line 102
         at MeshWeaver.AI.StreamStallGuardChatClient.GetStreamingResponseAsync(IEnumerable`1 messages, ChatOptions options, CancellationToken cancellationToken)+MoveNext() in /home/runner/work/MeshWeaver/MeshWeaver/plugins-repo/src/MeshWeaver.AI/StreamStallGuardChatClient.cs:line 188
         at MeshWeaver.AI.StreamStallGuardChatClient.GetStreamingResponseAsync(IEnumerable`1 messages, ChatOptions options, CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
         at MeshWeaver.AI.ProviderCallTimingChatClient.GetStreamingResponseAsync(IEnumerable`1 messages, ChatOptions options, CancellationToken cancellationToken)+MoveNext() in /home/runner/work/MeshWeaver/MeshWeaver/plugins-repo/src/MeshWeaver.AI/ProviderCallTimingChatClient.cs:line 64
         at MeshWeaver.AI.ProviderCallTimingChatClient.GetStreamingResponseAsync(IEnumerable`1 messages, ChatOptions options, CancellationToken cancellationToken)+MoveNext() in /home/runner/work/MeshWeaver/MeshWeaver/plugins-repo/src/MeshWeaver.AI/ProviderCallTimingChatClient.cs:line 108
         at MeshWeaver.AI.ProviderCallTimingChatClient.GetStreamingResponseAsync(IEnumerable`1 messages, ChatOptions options, CancellationToken cancellationToken)+System.Threading.Tasks.Sources.…[truncated]
2026-09-28 11:32:04Z memex-portal-deployment-5c476fdbf8-6hr6s fail: MeshWeaver.AI.AgentChatClient[0]
      [ThreadExec] ERROR: 11:32:04.036 threadPath=Hosting/Triage/_Thread/review-pull-request-systemorph-meshweave-c02c
      System.TimeoutException: AI streaming exceeded the maximum round duration of 00:30:00 and was aborted (#147). The model endpoint stopped responding mid-stream; resubmit to retry.
       ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
         at MeshWeaver.AI.StreamStallGuardChatClient.GetStreamingResponseAsync(IEnumerable`1 messages, ChatOptions options, CancellationToken cancellationToken)+MoveNext() in /home/runner/work/MeshWeaver/MeshWeaver/plugins-repo/src/MeshWeaver.AI/StreamStallGuardChatClient.cs:line 102
         at MeshWeaver.AI.StreamStallGuardChatClient.GetStreamingResponseAsync(IEnumerable`1 messages, ChatOptions options, CancellationToken cancellationToken)+MoveNext() in /home/runner/work/MeshWeaver/MeshWeaver/plugins-repo/src/MeshWeaver.AI/StreamStallGuardChatClient.cs:line 188
         at MeshWeaver.AI.StreamStallGuardChatClient.GetStreamingResponseAsync(IEnumerable`1 messages, ChatOptions options, CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
         at MeshWeaver.AI.ProviderCallTimingChatClient.GetStreamingResponseAsync(IEnumerable`1 messages, ChatOptions options, CancellationToken cancellationToken)+MoveNext() in /home/runner/work/MeshWeaver/MeshWeaver/plugins-repo/src/MeshWeaver.AI/ProviderCallTimingChatClient.cs:line 64
         at MeshWeaver.AI.ProviderCallTimingChatClient.GetStreamingResponseAsync(IEnumerable`1 messages, ChatOptions options, CancellationToken cancellationToken)+MoveNext() in /home/runner/work/MeshWeaver/MeshWeaver/plugins-repo/src/MeshWeaver.AI/ProviderCallTimingChatClient.cs:line 108
         at MeshWeaver.AI.ProviderCallTimingChatClient.GetStreamingResponseAsync(IEnumerable`1 messages, ChatOptions options, CancellationToken cancellationToken)+System.Threading.Tasks.Sources.…[truncated]
2026-09-28 12:24:28Z memex-portal-deployment-5c476fdbf8-6hr6s fail: MeshWeaver.AI.AgentChatClient[0]
      [ThreadExec] ERROR: 12:24:28.672 threadPath=Hosting/Triage/_Thread/review-pull-request-systemorph-meshweave-a2a4
      System.TimeoutException: AI streaming exceeded the maximum round duration of 00:30:00 and was aborted (#147). The model endpoint stopped responding mid-stream; resubmit to retry.
       ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
         at MeshWeaver.AI.StreamStallGuardChatClient.GetStreamingResponseAsync(IEnumerable`1 messages, ChatOptions options, CancellationToken cancellationToken)+MoveNext() in /home/runner/work/MeshWeaver/MeshWeaver/plugins-repo/src/MeshWeaver.AI/StreamStallGuardChatClient.cs:line 102
         at MeshWeaver.AI.StreamStallGuardChatClient.GetStreamingResponseAsync(IEnumerable`1 messages, ChatOptions options, CancellationToken cancellationToken)+MoveNext() in /home/runner/work/MeshWeaver/MeshWeaver/plugins-repo/src/MeshWeaver.AI/StreamStallGuardChatClient.cs:line 188
         at MeshWeaver.AI.StreamStallGuardChatClient.GetStreamingResponseAsync(IEnumerable`1 messages, ChatOptions options, CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
         at MeshWeaver.AI.ProviderCallTimingChatClient.GetStreamingResponseAsync(IEnumerable`1 messages, ChatOptions options, CancellationToken cancellationToken)+MoveNext() in /home/runner/work/MeshWeaver/MeshWeaver/plugins-repo/src/MeshWeaver.AI/ProviderCallTimingChatClient.cs:line 64
         at MeshWeaver.AI.ProviderCallTimingChatClient.GetStreamingResponseAsync(IEnumerable`1 messages, ChatOptions options, CancellationToken cancellationToken)+MoveNext() in /home/runner/work/MeshWeaver/MeshWeaver/plugins-repo/src/MeshWeaver.AI/ProviderCallTimingChatClient.cs:line 108
         at MeshWeaver.AI.ProviderCallTimingChatClient.GetStreamingResponseAsync(IEnumerable`1 messages, ChatOptions options, CancellationToken cancellationToken)+System.Threading.Tasks.Sources.…[truncated]

_{Opened automatically from Admin/_LogIncident/8bfa7322a2e2815c. Recurrences are folded into this issue rather than opening new ones.}
_{It also stands for the whole log site 0b5aeed57173c35d: other fingerprints of this site fold in here as comments rather than opening tickets of their own.}

Still occurring: 1 more since the last update (4 total).

Last seen: 2026-09-28 18:46:31Z
Pods: memex-portal-deployment-5c476fdbf8-6hr6s, memex-portal-deployment-85469d6f4b-lrxh7

Recent log lines

2026-09-28 11:32:04Z memex-portal-deployment-5c476fdbf8-6hr6s fail: MeshWeaver.AI.AgentChatClient[0]
      [ThreadExec] ERROR: 11:32:04.036 threadPath=Hosting/Triage/_Thread/review-pull-request-systemorph-meshweave-c02c
      System.TimeoutException: AI streaming exceeded the maximum round duration of 00:30:00 and was aborted (#147). The model endpoint stopped responding mid-stream; resubmit to retry.
       ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
         at MeshWeaver.AI.StreamStallGuardChatClient.GetStreamingResponseAsync(IEnumerable`1 messages, ChatOptions options, CancellationToken cancellationToken)+MoveNext() in /home/runner/work/MeshWeaver/MeshWeaver/plugins-repo/src/MeshWeaver.AI/StreamStallGuardChatClient.cs:line 102
         at MeshWeaver.AI.StreamStallGuardChatClient.GetStreamingResponseAsync(IEnumerable`1 messages, ChatOptions options, CancellationToken cancellationToken)+MoveNext() in /home/runner/work/MeshWeaver/MeshWeaver/plugins-repo/src/MeshWeaver.AI/StreamStallGuardChatClient.cs:line 188
         at MeshWeaver.AI.StreamStallGuardChatClient.GetStreamingResponseAsync(IEnumerable`1 messages, ChatOptions options, CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
         at MeshWeaver.AI.ProviderCallTimingChatClient.GetStreamingResponseAsync(IEnumerable`1 messages, ChatOptions options, CancellationToken cancellationToken)+MoveNext() in /home/runner/work/MeshWeaver/MeshWeaver/plugins-repo/src/MeshWeaver.AI/ProviderCallTimingChatClient.cs:line 64
         at MeshWeaver.AI.ProviderCallTimingChatClient.GetStreamingResponseAsync(IEnumerable`1 messages, ChatOptions options, CancellationToken cancellationToken)+MoveNext() in /home/runner/work/MeshWeaver/MeshWeaver/plugins-repo/src/MeshWeaver.AI/ProviderCallTimingChatClient.cs:line 108
         at MeshWeaver.AI.ProviderCallTimingChatClient.GetStreamingResponseAsync(IEnumerable`1 messages, ChatOptions options, CancellationToken cancellationToken)+System.Threading.Tasks.Sources.…[truncated]
2026-09-28 12:24:28Z memex-portal-deployment-5c476fdbf8-6hr6s fail: MeshWeaver.AI.AgentChatClient[0]
      [ThreadExec] ERROR: 12:24:28.672 threadPath=Hosting/Triage/_Thread/review-pull-request-systemorph-meshweave-a2a4
      System.TimeoutException: AI streaming exceeded the maximum round duration of 00:30:00 and was aborted (#147). The model endpoint stopped responding mid-stream; resubmit to retry.
       ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
         at MeshWeaver.AI.StreamStallGuardChatClient.GetStreamingResponseAsync(IEnumerable`1 messages, ChatOptions options, CancellationToken cancellationToken)+MoveNext() in /home/runner/work/MeshWeaver/MeshWeaver/plugins-repo/src/MeshWeaver.AI/StreamStallGuardChatClient.cs:line 102
         at MeshWeaver.AI.StreamStallGuardChatClient.GetStreamingResponseAsync(IEnumerable`1 messages, ChatOptions options, CancellationToken cancellationToken)+MoveNext() in /home/runner/work/MeshWeaver/MeshWeaver/plugins-repo/src/MeshWeaver.AI/StreamStallGuardChatClient.cs:line 188
         at MeshWeaver.AI.StreamStallGuardChatClient.GetStreamingResponseAsync(IEnumerable`1 messages, ChatOptions options, CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
         at MeshWeaver.AI.ProviderCallTimingChatClient.GetStreamingResponseAsync(IEnumerable`1 messages, ChatOptions options, CancellationToken cancellationToken)+MoveNext() in /home/runner/work/MeshWeaver/MeshWeaver/plugins-repo/src/MeshWeaver.AI/ProviderCallTimingChatClient.cs:line 64
         at MeshWeaver.AI.ProviderCallTimingChatClient.GetStreamingResponseAsync(IEnumerable`1 messages, ChatOptions options, CancellationToken cancellationToken)+MoveNext() in /home/runner/work/MeshWeaver/MeshWeaver/plugins-repo/src/MeshWeaver.AI/ProviderCallTimingChatClient.cs:line 108
         at MeshWeaver.AI.ProviderCallTimingChatClient.GetStreamingResponseAsync(IEnumerable`1 messages, ChatOptions options, CancellationToken cancellationToken)+System.Threading.Tasks.Sources.…[truncated]
2026-09-28 18:46:31Z memex-portal-deployment-85469d6f4b-lrxh7 fail: MeshWeaver.AI.AgentChatClient[0]
      [ThreadExec] ERROR: 18:46:31.468 threadPath=Hosting/Triage/_Thread/review-pull-request-systemorph-meshweave-dd21
      System.TimeoutException: AI streaming exceeded the maximum round duration of 00:30:00 and was aborted (#147). The model endpoint stopped responding mid-stream; resubmit to retry.
       ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
         at MeshWeaver.AI.StreamStallGuardChatClient.GetStreamingResponseAsync(IEnumerable`1 messages, ChatOptions options, CancellationToken cancellationToken)+MoveNext() in /home/runner/work/MeshWeaver/MeshWeaver/plugins-repo/src/MeshWeaver.AI/StreamStallGuardChatClient.cs:line 102
         at MeshWeaver.AI.StreamStallGuardChatClient.GetStreamingResponseAsync(IEnumerable`1 messages, ChatOptions options, CancellationToken cancellationToken)+MoveNext() in /home/runner/work/MeshWeaver/MeshWeaver/plugins-repo/src/MeshWeaver.AI/StreamStallGuardChatClient.cs:line 188
         at MeshWeaver.AI.StreamStallGuardChatClient.GetStreamingResponseAsync(IEnumerable`1 messages, ChatOptions options, CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
         at MeshWeaver.AI.ProviderCallTimingChatClient.GetStreamingResponseAsync(IEnumerable`1 messages, ChatOptions options, CancellationToken cancellationToken)+MoveNext() in /home/runner/work/MeshWeaver/MeshWeaver/plugins-repo/src/MeshWeaver.AI/ProviderCallTimingChatClient.cs:line 64
         at MeshWeaver.AI.ProviderCallTimingChatClient.GetStreamingResponseAsync(IEnumerable`1 messages, ChatOptions options, CancellationToken cancellationToken)+MoveNext() in /home/runner/work/MeshWeaver/MeshWeaver/plugins-repo/src/MeshWeaver.AI/ProviderCallTimingChatClient.cs:line 108
         at MeshWeaver.AI.ProviderCallTimingChatClient.GetStreamingResponseAsync(IEnumerable`1 messages, ChatOptions options, CancellationToken cancellationToken)+System.Threading.Tasks.Sources.…[truncated]

Exact duplicate of #5858: same fingerprint 8bfa7322a2e2815c, three timestamps (11:14:24Z, 11:32:04Z, 12:24:28Z), pod, stack and impact. Keeping #5858 open as the single tracking issue for the provider-stall cause and its user-visible outcome. No timeout increase or retry is being treated as a root-cause fix.