Agent thread execution fails on transient model-provider connection reset mid-SSE-stream #5892
Description
What is failing
During an agent thread execution (an automated PR-review triage thread), the model provider's streaming chat-completion response was cut off mid-stream by a TCP reset, and the exception propagated unhandled out of the SSE consumption loop into the ThreadExec error handler, which logged the run as failed. The component at the boundary is MeshWeaver.AI.AgentChatClient.
Probable cause
The reset itself is external — socket error 104 (ECONNRESET) from the remote peer (provider or an intermediary gateway closing a long-lived SSE connection), with the visible stack running entirely through the .NET networking stack and the OpenAI SDK's SSE enumerator (OpenAI.AsyncSseUpdateCollection1.AsyncSseUpdateEnumerator1...MoveNextAsync) and no MeshWeaver frame in between. High confidence on that mechanism; what is not established is whether AgentChatClient is supposed to retry or degrade gracefully on a transient stream failure. The exception escaping to the ThreadExec ERROR site suggests it does not, and that is the actionable defect. If a retry does exist and this was logged after it, then the issue is instead the log level — a transient external event surfaced as ERROR with a full stack.
Impact
Single occurrence on a single pod; one background agent run interrupted. No user-visible data loss and no portal impact — re-running the thread is the workaround. If these recur at any rate, the reliability of all automated thread executions is affected and the severity should be re-ruled.
Where to look
MeshWeaver.AI.AgentChatClient — the streaming-response consumption path around the OpenAI SDK's async streaming collection — and the ThreadExec error handler that emits the [ThreadExec] ERROR ... threadPath=... line. The MeshWeaver.AI source is not locatable in the readable mesh from this triage context, so start from the log category and the ThreadExec error site rather than a file link.
No existing incident or issue covers this component/symptom — this appears to be the first.
Evidence
| Fingerprint | a6273decbdc10005 |
| Category | MeshWeaver.AI.AgentChatClient |
| Severity | Error |
| Exception | System.IO.IOException |
| Top frame | OpenAI.AsyncSseUpdateCollection1.AsyncSseUpdateEnumerator1.System.Collections.Generic.IAsyncEnumerator<U>.MoveNextAsync() |
| Namespace | memex |
| Pods | memex-portal-deployment-5dc5995596-d9qj7 |
| Occurrences | 1 |
| First seen | 2026-09-29 09:26:32Z |
| Last seen | 2026-09-29 09:26:32Z |
| Routing | not determined — no configured route matches the category MeshWeaver.AI.AgentChatClient. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject. |
Recent log lines
2026-09-29 09:26:32Z memex-portal-deployment-5dc5995596-d9qj7 fail: MeshWeaver.AI.AgentChatClient[0]
[ThreadExec] ERROR: 09:26:32.677 threadPath=Hosting/Triage/_Thread/review-pull-request-systemorph-meshweave-668a
System.IO.IOException: Unable to read data from the transport connection: Connection reset by peer.
---> System.Net.Sockets.SocketException (104): Connection reset by peer
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.CreateException(SocketError error, Boolean forAsyncThrow)
at System.Net.Sockets.NetworkStream.ReadAsync(Memory`1 buffer, CancellationToken cancellationToken)
at System.Net.Security.SslStream.EnsureFullTlsFrameAsync[TIOAdapter](CancellationToken cancellationToken, Int32 estimatedSize)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Net.Sockets.SocketAsyncEngine.System.Threading.IThreadPoolWorkItem.Execute()
at System.Threading.ThreadPoolWorkQueue.Dispatch()
at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
at System.Threading.Thread.StartCallback()
--- End of stack trace from previous location ---
--- End of inner exception stack trace ---
at System.Net.Security.SslStream.EnsureFullTlsFrameAsync[TIOAdapter](CancellationToken cancellationToken, Int32 estimatedSize)
at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16 token)
at System.Net.Security.SslStream.ReadAsyncInternal[TIOAdapter](Memory`1 buffer, CancellationToken cancellationToken)
at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16 token)
at System.Net.Http.HttpConnection.FillAsync(Boolean async)
at System.Net.Http.HttpConnection.ChunkedEncodingRea…[truncated]
Opened automatically from Admin/_LogIncident/a6273decbdc10005. Recurrences are folded into this issue rather than opening new ones.
It also stands for the whole log site 85c3f064b5f7348e: other fingerprints of this site fold in here as comments rather than opening tickets of their own.
Activity
systemorph commentedon Oct 6, 2026
Still occurring: 4 more since the last update (5 total).
memex-portal-deployment-5dc5995596-d9qj7,memex-portal-deployment-65d864bddf-v5s6sRecent log lines
systemorph commentedon Oct 6, 2026
Still occurring: 1 more since the last update (6 total).
memex-portal-deployment-5dc5995596-d9qj7,memex-portal-deployment-65d864bddf-v5s6sRecent log lines
rbuergi commentedon Oct 7, 2026
Related: #5890. The provider-stream failure family on the
[ThreadExec]round loop (no automatic recovery); the full cross-link list is there. Not merged: different failure arm.rbuergi commentedon Oct 8, 2026
Fix in Systemorph/MeshWeaver.Plugins#3192. A peer reset arrives as
IOException(SocketException ConnectionReset), not asHttpIOException, soStreamStallGuardChatClient.FindTruncationnever classified it. The post-#2695 recurrence on 2026-10-06 06:11Z (five resets in 9 s on one pod, across 4 threads) therefore never got the before-output retry.ConnectionResetis now named a truncation: a reset before any output is retried, and a reset after output ends the round with a named cause. Every other socket error keeps its unclassified path. Tests:APeerResetBeforeAnyOutput_IsNamedATruncation, with the controlASocketFailureOtherThanAPeerReset_IsNotRelabelled. Related: #5890 (cut after output started).sev:L sweep, Claude Opus 5.5 (agent session).
rbuergi commentedon Oct 10, 2026
I checked this on memex with a governed
Logsread on control (Ops/Actions/logs-memex-20261010-w8b-provider-stream-faults,{namespace="memex"} |~ "(?i)((ProviderStreamException)|(Connection reset by peer))"). The window was 2026-10-07 10:03Z to 2026-10-10 10:03Z: 426 lines, of which 4 areConnection reset by peerresets (10-07 10:05Z and 10:19Z, 10-08 06:07Z and 08:07Z).All four came before MeshWeaver.Plugins#3192 merged (2026-10-08 20:26Z). There has been none since, so there is no post-fix occurrence to prove the classification live. I am leaving this open until the next reset shows
[ProviderStream] TRUNCATED (ConnectionReset …)with a RETRY or REGENERATE instead of a[ThreadExec] ERROR.MeshWeaver.Plugins#3289 extends the same path to a reset AFTER text: the call is regenerated with the partial answer withdrawn (
APeerResetAfterText_ThroughTheStallGuard_IsRegenerated).Claude Opus 5.5 (agent session, w8b).
rbuergi commentedon Oct 11, 2026
The retry policy this issue asks for is implemented and serving. Closing #5880, #5890 and #5892 together (one policy, three failure arms).
The rule, as it stands in
ProviderCallRetryChatClient(MeshWeaver.Plugins):[ProviderRetry] RETRY)[ProviderRetry] REGENERATE)It is bounded (retry count, base delay and a retry window) and classified by cause: a stall, a truncated body, a peer reset and a provider fault its guard marked transient (streamed 408/429/5xx,
finish_reason: "error") qualify; a streamed 402/403/404 does not. This is a designed retry of an idempotent step, not a raised bound: it sits below the function invoker, so one retry is one model call and never a re-run of a tool.Which change covers which arm:
IOException(SocketException ConnectionReset)): Plugins#3192 classifies it as a truncation, so it takes the same two paths.Tests on the repository's fake provider:
ProviderCallRetryRoundTest,ProviderStreamStallRoundTest,APeerResetBeforeAnyOutput_IsNamedATruncation,APeerResetAfterText_ThroughTheStallGuard_IsRegenerated.What is running. The control instance runs
memex-portal-ai:3.1.10517on all 3 replicas (SampleOps/Actions/sample-20261011-w14-ai, 2026-10-11 11:04Z). Itspluginspublication is sealed at42156f2a, which contains the merge commits of #2695, #3192 and #3289 (git merge-base --is-ancestorfor each).Readings, 2026-10-11:
LogsreadOps/Actions/logs-20261011-w14-d-ai-24h:[ProviderRetry],[ProviderStream],[ThreadExec] ERROR, 2026-10-10 11:19Z → 10-11 11:19Z: 0 lines, not cut. Positive control for the pattern:Ops/Actions/logs-20261011-w14-ctl-bracketmatched the two[ThreadExec] AUTH_REQUIREDlines of a probe round.2ab9a57b70c6c001, response ended) and noAgentChatClientincident of any kind has been updated since 2026-10-09 11:30Z. The four peer resets of 10-07/10-08 all predate DrainAll returned with a Layout permit still in flight — the teardown join window the #613 fix closes, standing open intermittently #3192.Not established: a retry observed live. No provider fault occurred in the 24 h I could read (a 72 h read was refused by Loki with 429), so there is no
[ProviderRetry]line to show. A recurrence is filed here automatically by the log watcher; what to read then is whether a[ProviderRetry] RETRYorREGENERATEwarning precedes the error. If it does, the round was cut after a tool call or the retry budget was spent, which is the designed outcome. If it does not, the classification missed that exception shape.wave w14, Claude Opus 5.5 (agent session).