Repository navigation
OpenAIWireStreamGuard mislabels the redaction filter's 403 PII block as a provider-account fault; redaction_context_lost aborts another agent round (recurrence of #5932) #5960
Description
What is failing
An agent round's streaming chat call (category MeshWeaver.AI.AgentChatClient) is rejected mid-round by the model gateway's PII redaction filter with 403 Request blocked: PII detected (redaction_context_lost), and the round aborts with nothing posted. On top of that, the ProviderStreamException thrown by OpenAIWireStreamGuard carries a canned message asserting this is "a fault at the provider or the provider account (a key limit, exhausted credit, or no upstream able to serve the request), not a problem with the request" — which is false for this failure class: a content-policy PII block is a request-side rejection, not an account fault.
Probable cause
High confidence on the mechanism, established for the identical prior burst in #5932: the gateway-side redaction filter cannot find/restore its per-request redaction context and applies a fail-closed policy (block rather than redact statelessly). This occurrence is a recurrence of that unfixed defect on a different pr-reviewer round.
Separately, high confidence (visible in the sample text itself): OpenAIWireStreamGuard classifies any non-success provider answer under the one "provider/account fault" canned message, so a 403 content-policy block is misdescribed in logs and to callers. The two should be fixed together: distinguish block-class errors (4xx content policy / redaction) from account/availability faults, and treat redaction_context_lost per #5932's fix direction (redact statelessly or re-establish context instead of failing closed).
Impact
Judged from the numbers: 1 occurrence in this burst, second known occurrence overall (two pr-reviewer rounds aborted across two days, one pod each). Each abort loses a completed analysis round (~work discarded, no node updated) — a secondary automated path with an easy workaround (re-run the round). Not release-blocking.
Known / duplicate status
Near-duplicate of #5932 (filed 2026-09-30 from incident 367de820103e2273, same component and same 403 redaction_context_lost; see also the related masking-mode issue MeshWeaver.Plugins#2567, same layer). This ticket adds the recurrence evidence and the error-classification defect in the guard's canned message; a human may close or merge it against #5932 in seconds.
Where to look
src/MeshWeaver.AI.OpenAI/OpenAIWireStreamGuard.cs—GetStreamingResponseAsync(exception built around line 85 per the sample stack): split the canned provider-fault message by error class.- The redaction filter's
redaction_context_lostreason-code emission site (per PII redaction filter on the model gateway blocks agent chat requests outright when its redaction context is lost, aborting agent rounds mid-flight #5932's diagnose record, keyed/stored/looked-up per request) for the fail-closed policy. - Mesh activity record: Governance/Activities/bug-diagnose-systemorph-meshweaver-5932.
Evidence
| Fingerprint | c715439554baaeab |
| Category | MeshWeaver.AI.AgentChatClient |
| Severity | Error |
| Exception | MeshWeaver.AI.ProviderStreamException |
| Top frame | MeshWeaver.AI.OpenAI.OpenAIWireStreamGuard.GetStreamingResponseAsync(IEnumerable1 messages, ChatOptions options, CancellationToken cancellationToken)+MoveNext()` |
| Namespace | memex |
| Pods | memex-portal-deployment-7975d75fb-zk5l2 |
| Occurrences | 1 |
| First seen | 2026-10-01 10:36:43Z |
| Last seen | 2026-10-01 10:36:43Z |
| Routing | not determined — no configured route matches the category MeshWeaver.AI.AgentChatClient. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject. |
Recent log lines
2026-10-01 10:36:43Z memex-portal-deployment-7975d75fb-zk5l2 fail: MeshWeaver.AI.AgentChatClient[0]
[ThreadExec] ERROR: 10:36:43.925 threadPath=Hosting/Triage/_Thread/review-pull-request-systemorph-meshweave-8fc3
MeshWeaver.AI.ProviderStreamException: The model provider answered with an error instead of a response: [403] Request blocked: PII detected (redaction_context_lost). This is a fault at the provider or the provider account (a key limit, exhausted credit, or no upstream able to serve the request), not a problem with the request.
at MeshWeaver.AI.OpenAI.OpenAIWireStreamGuard.GetStreamingResponseAsync(IEnumerable`1 messages, ChatOptions options, CancellationToken cancellationToken)+MoveNext() in /repo/src/MeshWeaver.AI.OpenAI/OpenAIWireStreamGuard.cs:line 85
at MeshWeaver.AI.OpenAI.OpenAIWireStreamGuard.GetStreamingResponseAsync(IEnumerable`1 messages, ChatOptions options, CancellationToken cancellationToken)+MoveNext() in /repo/src/MeshWeaver.AI.OpenAI/OpenAIWireStreamGuard.cs:line 112
at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
--- End of stack trace from previous location ---
at MeshWeaver.AI.StreamStallGuardChatClient.Swallow(Task task) in /home/runner/work/MeshWeaver/MeshWeaver/plugins-repo/src/MeshWeaver.AI/StreamStallGuardChatClient.cs:line 221
at MeshWeaver.AI.StreamStallGuardChatClient.GetStreamingResponseAsync(IEnumerable`1 messages, ChatOptions options, CancellationToken cancellationToken)+MoveNext() in /home/runner/work/MeshWeaver/MeshWeaver/plugins-repo/src/MeshWeaver.AI/StreamStallGuardChatClient.cs:line 188
at MeshWeaver.AI.StreamStallGuardChatClient.GetStreamingResponseAsync(IEnumerable`1 messages, ChatOptions options, CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
at MeshWeaver.AI.ProviderCallTimingChatClient.GetStreamingResponseAsync(IEnumerable`1 messages, ChatOptions options, CancellationToken cancellationTok…[truncated]
Opened automatically from Admin/_LogIncident/c715439554baaeab. Recurrences are folded into this issue rather than opening new ones.
It also stands for the whole log site dc3faa884909b1cd: other fingerprints of this site fold in here as comments rather than opening tickets of their own.
Activity
systemorph commentedon Oct 1, 2026
Still occurring: 0 more since the last update (1 total).
memex-portal-deployment-7975d75fb-zk5l2Recent log lines
systemorph commentedon Oct 1, 2026
Folded in — a DIFFERENT fingerprint from the same log site.
This issue tracks the log site
dc3faa884909b1cd(MeshWeaver.AI.AgentChatClient), and these occurrences belong to it. They carry their own fingerprint — the masking split one site across several — so before this they would have opened a SEPARATE ticket. One log site fanned out into 46 of those, which is why they fold here instead. Nothing is discarded: the distinguishing evidence is below, and splitting this back out is a person's call.42ea053aa2f12f04dc3faa884909b1cdMeshWeaver.AI.AgentChatClientMeshWeaver.AI.ProviderStreamExceptionMeshWeaver.AI.OpenAI.OpenAIWireStreamGuard.GetStreamingResponseAsync(IEnumerable1 messages, ChatOptions options, CancellationToken cancellationToken)+MoveNext()`The model provider answered with an error instead of a response: [{n}] Request blocked: PII detected (arguments_repair_unavailable). This is a fault at the provider or the provider account (a key limit, exhausted credit, or no upstream able to serve the request), not a problem with the request.memexmemex-portal-deployment-7975d75fb-zk5l2Admin/_LogIncident/42ea053aa2f12f04arguments_repair_unavailable, aborting PR-review rounds mid-flightWhat triage wrote about this fingerprint
What is failing
The PII redaction layer in front of the model gateway rejects an outbound chat-completion request from
MeshWeaver.AI.AgentChatClientoutright with HTTP 403 (Request blocked: PII detected (arguments_repair_unavailable)) instead of redacting and forwarding it. TheProviderStreamExceptionsurfaces inside the agent's streaming call and aborts the whole agent round — in this case a pr-reviewer round on threadHosting/Triage/_Thread/review-pull-request-systemorph-meshweave-6fa6, whose review was lost mid-flight.Probable cause
This is the same fail-closed redaction filter already diagnosed in #5932, but under a new reason code:
arguments_repair_unavailablerather thanredaction_context_lost. That code suggests the filter detected PII in (tool-call) arguments, attempted its arguments-repair/redaction path, found it unavailable, and blocked the request rather than sending unredacted content. Moderate confidence on the mechanism (the reason code names it directly); the reason the repair path was unavailable is not established by the evidence. Note also that the exception's fixed attribution sentence ("a fault at the provider or the provider account… not a problem with the request") is false here — that defect is already filed as #5960 and applies to this burst too.Impact
A single occurrence on one pod so far, but the failure mode is expensive when it fires: the entire agent round is lost, including tool work already done, and the review is silently not posted. Same blast radius as the three
redaction_context_lostoccurrences tracked in #5932 — if both reason codes keep firing, every long-lived agent round is at risk. Workaround: re-run the round. Severity left at M consistent with #5932; a human may reasonably fold this into #5932 as another symptom of the same filter.Where to look
The gateway-side PII redaction middleware behind
MeshWeaver.AI.AgentChatClient(per #5932's diagnosis:src/MeshWeaver.AI/OpenAI/PiiRedactionFilter.cs). Start from the code that emitsarguments_repair_unavailable: where the arguments-repair path is keyed and looked up, what makes it report "unavailable", and why that degrades to a hard block instead of a stateless redaction or a retry. Related, same component: #5932 (redaction_context_lostblocking, still open) and Systemorph/MeshWeaver.Plugins#2567 (over-masking sibling); #5960 covers the misleading attribution sentence that this burst also exhibits.Still occurring: 1 more since the last update (1 total).
memex-portal-deployment-7975d75fb-zk5l2Recent log lines
systemorph commentedon Oct 1, 2026
Folded in — a DIFFERENT fingerprint from the same log site.
This issue tracks the log site
dc3faa884909b1cd(MeshWeaver.AI.AgentChatClient), and these occurrences belong to it. They carry their own fingerprint — the masking split one site across several — so before this they would have opened a SEPARATE ticket. One log site fanned out into 46 of those, which is why they fold here instead. Nothing is discarded: the distinguishing evidence is below, and splitting this back out is a person's call.a7fbdce246651c00dc3faa884909b1cdMeshWeaver.AI.AgentChatClientMeshWeaver.AI.ProviderStreamExceptionMeshWeaver.AI.OpenAI.OpenAIWireStreamGuard.GetStreamingResponseAsync(IEnumerable1 messages, ChatOptions options, CancellationToken cancellationToken)+MoveNext()`The model provider answered with an error instead of a response: [{n}] Upstream error from Inceptron: EngineCore encountered an issue. See stack trace (above) for the root cause.. This is a fault at the provider or the provider account (a key limit, exhausted credit, or no upstream able to serve the request), not a problem with the request.memexmemex-portal-deployment-7975d75fb-zk5l2Admin/_LogIncident/a7fbdce246651c00What triage wrote about this fingerprint
What is failing
The upstream model provider (Inceptron) answered two agent-round streaming requests with
[502] Upstream error … EngineCore encountered an issue, andOpenAIWireStreamGuardcorrectly surfaced this as aProviderStreamException— aborting both PR-review rounds with no retry. Unlike the redaction-block shapes at this log site (#5960, #5932), the guard's canned "fault at the provider, not a problem with the request" message is accurate here; the defect is that a transient upstream 5xx costs the whole round.Probable cause
High confidence that the trigger is an upstream/provider-side fault (the 502 is reported verbatim by the provider and two independent rounds failed within milliseconds of each other on the same pod — a provider blip, not a per-request problem). The product-side defect is the already-documented retry gap: transient provider errors (connection resets #5892, prematurely-ended streams #5890, stalls #5858/#5859, and now 5xx error answers) abort agent rounds with no automatic retry, discarding completed analysis work.
Impact
Two PR-review agent rounds lost in one burst, single pod. Secondary automated path with an easy workaround (re-run the round) — but this is at least the fifth distinct transient-failure shape from this provider pipeline, each costing a completed round. Not release-blocking.
Where to look
src/MeshWeaver.AI.OpenAI/OpenAIWireStreamGuard.cs—GetStreamingResponseAsync(exception raised around line 85 per the stack): this is where a 5xx-class provider answer could be classified as retryable and re-issued before it propagates throughStreamStallGuardChatClient(src/MeshWeaver.AI/StreamStallGuardChatClient.cs, lines 188/221). A shared retry policy for transient provider faults would close this and the #5890/#5892 family together.Related, not duplicates
Still occurring: 2 more since the last update (2 total).
memex-portal-deployment-7975d75fb-zk5l2Recent log lines
systemorph commentedon Oct 1, 2026
Folded in — a DIFFERENT fingerprint from the same log site.
This issue tracks the log site
dc3faa884909b1cd(MeshWeaver.AI.AgentChatClient), and these occurrences belong to it. They carry their own fingerprint — the masking split one site across several — so before this they would have opened a SEPARATE ticket. One log site fanned out into 46 of those, which is why they fold here instead. Nothing is discarded: the distinguishing evidence is below, and splitting this back out is a person's call.8eeae8b9a3166b10dc3faa884909b1cdMeshWeaver.AI.AgentChatClientMeshWeaver.AI.ProviderStreamExceptionMeshWeaver.AI.OpenAI.OpenAIWireStreamGuard.GetStreamingResponseAsync(IEnumerable1 messages, ChatOptions options, CancellationToken cancellationToken)+MoveNext()`The model provider answered with an error instead of a response: [{n}] Network connection lost.. This is a fault at the provider or the provider account (a key limit, exhausted credit, or no upstream able to serve the request), not a problem with the request.memexmemex-portal-deployment-6b555484b4-qjm86Admin/_LogIncident/8eeae8b9a3166b10What triage wrote about this fingerprint
What is failing
The model provider answered two agent-round streaming requests with
[502] Network connection lost, andOpenAIWireStreamGuardsurfaced it as aProviderStreamExceptionthat aborted both PR-review rounds with no retry. As with the earlier 502 burst at this site, the guard's canned "fault at the provider, not a problem with the request" attribution is accurate for this error class — the defect is that a transient upstream 5xx costs the whole round.Probable cause
High confidence the trigger is a provider/upstream availability fault: the 502 is reported verbatim by the provider, two independent rounds failed within milliseconds of each other on the same pod, and this is the second 502 shape at this site within ~2 hours (the 11:01Z burst,
a7fbdce246651c00, was[502] Upstream error from Inceptron). The product-side defect is the retry gap already documented for this pipeline: transient provider faults (connection resets #5892, prematurely-ended streams #5890, and 5xx error answers) abort rounds and discard completed analysis work instead of being retried.Impact
Two automated PR-review rounds lost in one burst, single pod; re-running the round is the workaround. This is now the second 502 burst of the day, suggesting elevated upstream instability today — worth watching the recurrence rate, but the numbers are still nuisance-level, not an outage.
Where to look
src/MeshWeaver.AI.OpenAI/OpenAIWireStreamGuard.cs—GetStreamingResponseAsync(exception raised around line 85 per the stack), reached throughStreamStallGuardChatClient(src/MeshWeaver.AI/StreamStallGuardChatClient.cs, lines 188/221). A retryable classification for 5xx-class provider answers, shared with the #5890/#5892 family, closes this shape.Known / duplicate status
This log site (
dc3faa884909b1cd) stands for Systemorph/MeshWeaver#5960 (the redaction-403 shape); this fingerprint is the 5xx error class and near-duplicate of the 11:01Z 502 burst folded there asa7fbdce246651c00. Per the site rule it folds in as a comment; the retry-gap family to consolidate against is #5890/#5892. A human may merge or close against either in seconds.Still occurring: 2 more since the last update (2 total).
memex-portal-deployment-6b555484b4-qjm86Recent log lines
systemorph commentedon Oct 1, 2026
Folded in — a DIFFERENT fingerprint from the same log site.
This issue tracks the log site
dc3faa884909b1cd(MeshWeaver.AI.AgentChatClient), and these occurrences belong to it. They carry their own fingerprint — the masking split one site across several — so before this they would have opened a SEPARATE ticket. One log site fanned out into 46 of those, which is why they fold here instead. Nothing is discarded: the distinguishing evidence is below, and splitting this back out is a person's call.95d8ef000905325ddc3faa884909b1cdMeshWeaver.AI.AgentChatClientMeshWeaver.AI.ProviderStreamExceptionMeshWeaver.AI.OpenAI.OpenAIWireStreamGuard.GetStreamingResponseAsync(IEnumerable1 messages, ChatOptions options, CancellationToken cancellationToken)+MoveNext()`The model provider answered with an error instead of a response: [{n}] Provider returned error. This is a fault at the provider or the provider account (a key limit, exhausted credit, or no upstream able to serve the request), not a problem with the request.memexmemex-portal-deployment-555f74496d-k6vsxAdmin/_LogIncident/95d8ef000905325dWhat triage wrote about this fingerprint
What is failing
The model provider answered a streaming agent-round request with
429 Provider returned error, andOpenAIWireStreamGuardsurfaced it as aMeshWeaver.AI.ProviderStreamExceptionthat aborted the round with no retry — the round it killed was itself a log-incident triage round (Admin/_LogIncident/…/triage-the-log-incident…). This is the rate-limit shape of the already-documented retry gap: a transient provider fault costs the whole agent round, including any tool work already done.Probable cause
High confidence the trigger is a provider/account-side rate limit — the 429 is reported verbatim and the guard's canned "fault at the provider or the provider account… not a problem with the request" attribution is accurate for this error class (unlike the 403-PII shapes, where it is wrong — that message defect is filed as #5960). The product-side defect is the missing retry path: transient provider answers (429 rate limits, 5xx error answers, connection resets, prematurely-ended streams) are not classified as retryable, so the first hit fails the round.
Impact
One occurrence, one pod — nuisance level today, with an easy workaround (re-run the round). But this is the second 429 observed today and the fifth-plus distinct transient-failure shape from this provider pipeline, so each provider blip keeps costing a completed round until a shared retry policy exists.
Where to look
src/MeshWeaver.AI.OpenAI/OpenAIWireStreamGuard.cs—GetStreamingResponseAsync, where the error answer is wrapped (around line 85 per the stack), reached throughStreamStallGuardChatClient(src/MeshWeaver.AI/StreamStallGuardChatClient.cs, lines 188/221). Classify 429 (honouringRetry-Afterwhere present) and 5xx answers as retryable and re-issue with backoff before the exception propagates — this closes the whole transient-provider-fault family together.Known / duplicate status
Near-duplicate of the 06:43Z 429 today (incident
Admin/_LogIncident/0b9d6eb8fda3298a, filed as Systemorph/MeshWeaver#5932), which escaped as a rawClientResultExceptionthrough the non-streaming completion path; this fingerprint is the same defect surfaced through the wire-stream guard's wrapping. Also related, same family: the 502 bursts at this log site today and the #5890/#5892 retry-gap tickets. A human may merge this against #5932 in seconds. The site-folddc3faa884909b1cdstands for #5960; this error class is different from that ticket's subject, so treat this as a comment there, not a duplicate of it.Still occurring: 1 more since the last update (1 total).
memex-portal-deployment-555f74496d-k6vsxRecent log lines
systemorph commentedon Oct 1, 2026
Still occurring: 6 more since the last update (7 total).
memex-portal-deployment-7975d75fb-zk5l2,memex-portal-deployment-6b555484b4-4frk5,memex-portal-deployment-6b555484b4-qjm86,memex-portal-deployment-555f74496d-4hpzsRecent log lines
systemorph commentedon Oct 1, 2026
Still occurring: 1 more since the last update (2 total).
memex-portal-deployment-7975d75fb-zk5l2Recent log lines
systemorph commentedon Oct 1, 2026
Still occurring: 1 more since the last update (3 total).
memex-portal-deployment-7975d75fb-zk5l2,memex-portal-deployment-6f8fcf49d6-2nmsqRecent log lines
systemorph commentedon Oct 1, 2026
Still occurring: 1 more since the last update (2 total).
memex-portal-deployment-555f74496d-k6vsx,memex-portal-deployment-569bd77d9f-445h8Recent log lines
systemorph commentedon Oct 1, 2026
Still occurring: 2 more since the last update (9 total).
memex-portal-deployment-7975d75fb-zk5l2,memex-portal-deployment-6b555484b4-4frk5,memex-portal-deployment-6b555484b4-qjm86,memex-portal-deployment-555f74496d-4hpzs,memex-portal-deployment-84f9d886b4-25wlf,memex-portal-deployment-6f8fcf49d6-2nmsqRecent log lines
systemorph commentedon Oct 1, 2026
Still occurring: 2 more since the last update (9 total).
memex-portal-deployment-7975d75fb-zk5l2,memex-portal-deployment-6b555484b4-4frk5,memex-portal-deployment-6b555484b4-qjm86,memex-portal-deployment-555f74496d-4hpzs,memex-portal-deployment-84f9d886b4-25wlf,memex-portal-deployment-6f8fcf49d6-2nmsqRecent log lines
systemorph commentedon Oct 1, 2026
Still occurring: 1 more since the last update (3 total).
memex-portal-deployment-6b555484b4-qjm86,memex-portal-deployment-6f6c5b9978-mqvqzRecent log lines
systemorph commentedon Oct 2, 2026
Still occurring: 3 more since the last update (6 total).
memex-portal-deployment-7975d75fb-zk5l2,memex-portal-deployment-6f8fcf49d6-2nmsq,memex-portal-deployment-7b5dfddb9b-t2b8qRecent log lines
103 remaining items
systemorph commentedon Oct 7, 2026
Still occurring: 26 more since the last update (65 total).
memex-portal-deployment-75f7dd4cfb-mk985,memex-portal-deployment-67c67b456d-l7dp7,memex-portal-deployment-67c67b456d-255rq,memex-portal-deployment-5bbc8b6d-7rpnz,memex-portal-deployment-6485bffdff-cgd5r,memex-portal-deployment-848f664fd5-jgkbf,memex-portal-deployment-6cfcf78898-kss8q,memex-portal-deployment-6cfcf78898-lsklh,memex-portal-deployment-f86c77f48-45tjv,memex-portal-deployment-f86c77f48-5fmjr,memex-portal-deployment-65d864bddf-v5s6s,memex-portal-deployment-6bcdbb54fc-g9gh4,memex-portal-deployment-6f9c87b5c8-fsb5b,memex-portal-deployment-67dbbcf9b8-fnhhx,memex-portal-deployment-67dbbcf9b8-dv76t,memex-portal-deployment-7c9d678dd-zbgqc,memex-portal-deployment-7c9d678dd-wrgpn,memex-portal-deployment-54ff68d87d-f5hp9,memex-portal-deployment-5d74648df-ht6qx,memex-portal-deployment-5d74648df-ctdtsRecent log lines
systemorph commentedon Oct 7, 2026
Still occurring: 1 more since the last update (8 total).
memex-portal-deployment-7c4764b686-tbr7f,memex-portal-deployment-58475df96-nprkt,memex-portal-deployment-f86c77f48-45tjv,memex-portal-deployment-f86c77f48-g6pwj,memex-portal-deployment-65d864bddf-v5s6s,memex-portal-deployment-6bcdbb54fc-g9gh4,memex-portal-deployment-67dbbcf9b8-dv76tRecent log lines
systemorph commentedon Oct 7, 2026
Still occurring: 4 more since the last update (52 total).
memex-portal-deployment-7975d75fb-zk5l2,memex-portal-deployment-6f8fcf49d6-2nmsq,memex-portal-deployment-7b5dfddb9b-t2b8q,memex-portal-deployment-df798b484-cr6jn,memex-portal-deployment-5fb6c84b7-wc68k,memex-portal-deployment-5fb6c84b7-hlk7z,memex-portal-deployment-5fb6c84b7-8hhjb,memex-portal-deployment-7fc8c5f599-69jgf,memex-portal-deployment-5d99755fbf-mkpmk,memex-portal-deployment-7dd57d4d64-c5rv8,memex-portal-deployment-7f48f9cc6f-4rhtp,memex-portal-deployment-95f686766-v8qdn,memex-portal-deployment-6bd968f77b-6745f,memex-portal-deployment-559bc544dc-4wnc8,memex-portal-deployment-84b95bc48b-vjn9r,memex-portal-deployment-64796f4f56-b7hwc,memex-portal-deployment-64796f4f56-dfp92,memex-portal-deployment-75f7dd4cfb-mk985,memex-portal-deployment-67c67b456d-l7dp7,memex-portal-deployment-67c67b456d-s7nt9,memex-portal-deployment-67c67b456d-255rq,memex-portal-deployment-848f664fd5-zqfrh,memex-portal-deployment-848f664fd5-jgkbf,memex-portal-deployment-5dcbbd8f4b-w5g96,memex-portal-deployment-5dcbbd8f4b-jqqxx,memex-portal-deployment-58475df96-nprkt,memex-portal-deployment-6cfcf78898-kss8q,memex-portal-deployment-6cfcf78898-gkkdv,memex-portal-deployment-6cfcf78898-lsklh,memex-portal-deployment-f86c77f48-5fmjr,memex-portal-deployment-65d864bddf-v5s6s,memex-portal-deployment-6bcdbb54fc-g9gh4,memex-portal-deployment-67dbbcf9b8-kmlj6,memex-portal-deployment-67dbbcf9b8-t5c4w,memex-portal-deployment-67dbbcf9b8-fnhhx,memex-portal-deployment-7c9d678dd-wrgpnRecent log lines
systemorph commentedon Oct 7, 2026
Still occurring: 10 more since the last update (133 total).
memex-portal-deployment-7975d75fb-zk5l2,memex-portal-deployment-6b555484b4-4frk5,memex-portal-deployment-6b555484b4-qjm86,memex-portal-deployment-555f74496d-4hpzs,memex-portal-deployment-84f9d886b4-25wlf,memex-portal-deployment-6f8fcf49d6-2nmsq,memex-portal-deployment-6f6c5b9978-mqvqz,memex-portal-deployment-7b5dfddb9b-t2b8q,memex-portal-deployment-5758c659d8-tnlq2,memex-portal-deployment-5758c659d8-59kst,memex-portal-deployment-df798b484-cr6jn,memex-portal-deployment-df798b484-hcftm,memex-portal-deployment-58b948b4b6-xdrrl,memex-portal-deployment-66b8f9d689-h9mhd,memex-portal-deployment-7fc8c5f599-q4d8d,memex-portal-deployment-7fc8c5f599-69jgf,memex-portal-deployment-59f87cfd98-br5b9,memex-portal-deployment-5d99755fbf-mkpmk,memex-portal-deployment-7668648456-kwbfd,memex-portal-deployment-67b688cf6f-bndxb,memex-portal-deployment-67b688cf6f-sl57n,memex-portal-deployment-7f48f9cc6f-4rhtp,memex-portal-deployment-c8fbdbb5d-5gtdg,memex-portal-deployment-95f686766-v8qdn,memex-portal-deployment-6bd968f77b-6745f,memex-portal-deployment-6cf69869dc-h89vr,memex-portal-deployment-584d96cdb6-wr8ck,memex-portal-deployment-57d6d7f9cc-tslft,memex-portal-deployment-7b74d6bfb7-gwdkb,memex-portal-deployment-57f96dbd4d-ccssf,memex-portal-deployment-84b95bc48b-vjn9r,memex-portal-deployment-6dbcd749fb-2wm58,memex-portal-deployment-6dbcd749fb-ps2xz,memex-portal-deployment-6dbcd749fb-rxrg4,memex-portal-deployment-fc89c9bd8-pvjxz,memex-portal-deployment-75f7dd4cfb-lp6fq,memex-portal-deployment-75f7dd4cfb-mk985,memex-portal-deployment-67c67b456d-s7nt9,memex-portal-deployment-67c67b456d-255rq,memex-portal-deployment-f7664dd89-fnfrt,memex-portal-deployment-d7794fbdc-8ht27,memex-portal-deployment-848f664fd5-zqfrh,memex-portal-deployment-848f664fd5-jgkbf,memex-portal-deployment-5dcbbd8f4b-6kzlg,memex-portal-deployment-5dcbbd8f4b-w5g96,memex-portal-deployment-5dcbbd8f4b-jqqxx,memex-portal-deployment-7c4764b686-tbr7f,memex-portal-deployment-58475df96-nprkt,memex-portal-deployment-58475df96-vktt7,memex-portal-deployment-6cfcf78898-gkkdv,memex-portal-deployment-6cfcf78898-kss8q,memex-portal-deployment-f86c77f48-5fmjr,memex-portal-deployment-65d864bddf-v5s6s,memex-portal-deployment-6f9c87b5c8-fsb5b,memex-portal-deployment-6bcdbb54fc-g9gh4,memex-portal-deployment-6f9c87b5c8-p7xxp,memex-portal-deployment-67dbbcf9b8-kmlj6,memex-portal-deployment-7c9d678dd-wrgpn,memex-portal-deployment-54ff68d87d-f5hp9,memex-portal-deployment-766b9b4556-jfxzqRecent log lines
systemorph commentedon Oct 7, 2026
Still occurring: 1 more since the last update (66 total).
memex-portal-deployment-75f7dd4cfb-mk985,memex-portal-deployment-67c67b456d-l7dp7,memex-portal-deployment-67c67b456d-255rq,memex-portal-deployment-5bbc8b6d-7rpnz,memex-portal-deployment-6485bffdff-cgd5r,memex-portal-deployment-848f664fd5-jgkbf,memex-portal-deployment-6cfcf78898-kss8q,memex-portal-deployment-6cfcf78898-lsklh,memex-portal-deployment-f86c77f48-45tjv,memex-portal-deployment-f86c77f48-5fmjr,memex-portal-deployment-65d864bddf-v5s6s,memex-portal-deployment-6bcdbb54fc-g9gh4,memex-portal-deployment-6f9c87b5c8-fsb5b,memex-portal-deployment-67dbbcf9b8-fnhhx,memex-portal-deployment-67dbbcf9b8-dv76t,memex-portal-deployment-7c9d678dd-zbgqc,memex-portal-deployment-7c9d678dd-wrgpn,memex-portal-deployment-54ff68d87d-f5hp9,memex-portal-deployment-5d74648df-ht6qx,memex-portal-deployment-5d74648df-ctdts,memex-portal-deployment-766b9b4556-gqm6xRecent log lines
systemorph commentedon Oct 8, 2026
Still occurring: 1 more since the last update (134 total).
memex-portal-deployment-7975d75fb-zk5l2,memex-portal-deployment-6b555484b4-4frk5,memex-portal-deployment-6b555484b4-qjm86,memex-portal-deployment-555f74496d-4hpzs,memex-portal-deployment-84f9d886b4-25wlf,memex-portal-deployment-6f8fcf49d6-2nmsq,memex-portal-deployment-6f6c5b9978-mqvqz,memex-portal-deployment-7b5dfddb9b-t2b8q,memex-portal-deployment-5758c659d8-tnlq2,memex-portal-deployment-5758c659d8-59kst,memex-portal-deployment-df798b484-cr6jn,memex-portal-deployment-df798b484-hcftm,memex-portal-deployment-58b948b4b6-xdrrl,memex-portal-deployment-66b8f9d689-h9mhd,memex-portal-deployment-7fc8c5f599-q4d8d,memex-portal-deployment-7fc8c5f599-69jgf,memex-portal-deployment-59f87cfd98-br5b9,memex-portal-deployment-5d99755fbf-mkpmk,memex-portal-deployment-7668648456-kwbfd,memex-portal-deployment-67b688cf6f-bndxb,memex-portal-deployment-67b688cf6f-sl57n,memex-portal-deployment-7f48f9cc6f-4rhtp,memex-portal-deployment-c8fbdbb5d-5gtdg,memex-portal-deployment-95f686766-v8qdn,memex-portal-deployment-6bd968f77b-6745f,memex-portal-deployment-6cf69869dc-h89vr,memex-portal-deployment-584d96cdb6-wr8ck,memex-portal-deployment-57d6d7f9cc-tslft,memex-portal-deployment-7b74d6bfb7-gwdkb,memex-portal-deployment-57f96dbd4d-ccssf,memex-portal-deployment-84b95bc48b-vjn9r,memex-portal-deployment-6dbcd749fb-2wm58,memex-portal-deployment-6dbcd749fb-ps2xz,memex-portal-deployment-6dbcd749fb-rxrg4,memex-portal-deployment-fc89c9bd8-pvjxz,memex-portal-deployment-75f7dd4cfb-lp6fq,memex-portal-deployment-75f7dd4cfb-mk985,memex-portal-deployment-67c67b456d-s7nt9,memex-portal-deployment-67c67b456d-255rq,memex-portal-deployment-f7664dd89-fnfrt,memex-portal-deployment-d7794fbdc-8ht27,memex-portal-deployment-848f664fd5-zqfrh,memex-portal-deployment-848f664fd5-jgkbf,memex-portal-deployment-5dcbbd8f4b-6kzlg,memex-portal-deployment-5dcbbd8f4b-w5g96,memex-portal-deployment-5dcbbd8f4b-jqqxx,memex-portal-deployment-7c4764b686-tbr7f,memex-portal-deployment-58475df96-nprkt,memex-portal-deployment-58475df96-vktt7,memex-portal-deployment-6cfcf78898-gkkdv,memex-portal-deployment-6cfcf78898-kss8q,memex-portal-deployment-f86c77f48-5fmjr,memex-portal-deployment-65d864bddf-v5s6s,memex-portal-deployment-6f9c87b5c8-fsb5b,memex-portal-deployment-6bcdbb54fc-g9gh4,memex-portal-deployment-6f9c87b5c8-p7xxp,memex-portal-deployment-67dbbcf9b8-kmlj6,memex-portal-deployment-7c9d678dd-wrgpn,memex-portal-deployment-54ff68d87d-f5hp9,memex-portal-deployment-766b9b4556-jfxzq,memex-portal-deployment-d5cff4445-xkhq2Recent log lines
systemorph commentedon Oct 8, 2026
Still occurring: 1 more since the last update (135 total).
memex-portal-deployment-7975d75fb-zk5l2,memex-portal-deployment-6b555484b4-4frk5,memex-portal-deployment-6b555484b4-qjm86,memex-portal-deployment-555f74496d-4hpzs,memex-portal-deployment-84f9d886b4-25wlf,memex-portal-deployment-6f8fcf49d6-2nmsq,memex-portal-deployment-6f6c5b9978-mqvqz,memex-portal-deployment-7b5dfddb9b-t2b8q,memex-portal-deployment-5758c659d8-tnlq2,memex-portal-deployment-5758c659d8-59kst,memex-portal-deployment-df798b484-cr6jn,memex-portal-deployment-df798b484-hcftm,memex-portal-deployment-58b948b4b6-xdrrl,memex-portal-deployment-66b8f9d689-h9mhd,memex-portal-deployment-7fc8c5f599-q4d8d,memex-portal-deployment-7fc8c5f599-69jgf,memex-portal-deployment-59f87cfd98-br5b9,memex-portal-deployment-5d99755fbf-mkpmk,memex-portal-deployment-7668648456-kwbfd,memex-portal-deployment-67b688cf6f-bndxb,memex-portal-deployment-67b688cf6f-sl57n,memex-portal-deployment-7f48f9cc6f-4rhtp,memex-portal-deployment-c8fbdbb5d-5gtdg,memex-portal-deployment-95f686766-v8qdn,memex-portal-deployment-6bd968f77b-6745f,memex-portal-deployment-6cf69869dc-h89vr,memex-portal-deployment-584d96cdb6-wr8ck,memex-portal-deployment-57d6d7f9cc-tslft,memex-portal-deployment-7b74d6bfb7-gwdkb,memex-portal-deployment-57f96dbd4d-ccssf,memex-portal-deployment-84b95bc48b-vjn9r,memex-portal-deployment-6dbcd749fb-2wm58,memex-portal-deployment-6dbcd749fb-ps2xz,memex-portal-deployment-6dbcd749fb-rxrg4,memex-portal-deployment-fc89c9bd8-pvjxz,memex-portal-deployment-75f7dd4cfb-lp6fq,memex-portal-deployment-75f7dd4cfb-mk985,memex-portal-deployment-67c67b456d-s7nt9,memex-portal-deployment-67c67b456d-255rq,memex-portal-deployment-f7664dd89-fnfrt,memex-portal-deployment-d7794fbdc-8ht27,memex-portal-deployment-848f664fd5-zqfrh,memex-portal-deployment-848f664fd5-jgkbf,memex-portal-deployment-5dcbbd8f4b-6kzlg,memex-portal-deployment-5dcbbd8f4b-w5g96,memex-portal-deployment-5dcbbd8f4b-jqqxx,memex-portal-deployment-7c4764b686-tbr7f,memex-portal-deployment-58475df96-nprkt,memex-portal-deployment-58475df96-vktt7,memex-portal-deployment-6cfcf78898-gkkdv,memex-portal-deployment-6cfcf78898-kss8q,memex-portal-deployment-f86c77f48-5fmjr,memex-portal-deployment-65d864bddf-v5s6s,memex-portal-deployment-6f9c87b5c8-fsb5b,memex-portal-deployment-6bcdbb54fc-g9gh4,memex-portal-deployment-6f9c87b5c8-p7xxp,memex-portal-deployment-67dbbcf9b8-kmlj6,memex-portal-deployment-7c9d678dd-wrgpn,memex-portal-deployment-54ff68d87d-f5hp9,memex-portal-deployment-766b9b4556-jfxzq,memex-portal-deployment-d5cff4445-xkhq2,memex-portal-deployment-565c98f454-vlsjzRecent log lines
rbuergi commentedon Oct 8, 2026
The guard half is fixed in Systemorph/MeshWeaver.Plugins#3179. The gateway half is #5932.
Root cause.
src/MeshWeaver.AI.OpenAI/OpenAIWireStreamGuard.csDescribe(ProviderErrorChunk)attached one sentence to every streamed error object. A 403 is both a key limit and a gateway content block, so the code alone cannot classify it.Fix.
Describe(code, message)reads the wording:Request blocked,PII detected,flaggedormoderation→ "the model gateway refused this request under a content policy … not a fault of the provider, its key or its account";A block stays non-transient.
Tests.
GuardedClient_OnARedactionBlock_NamesARequestBlock_NotAnAccountFaultstreams the production payload.EachStreamedErrorClass_GetsItsOwnSentencehas 9 rows, with a 403 key limit and a 5xx "blocked" as controls. The fullMeshWeaver.AI.OpenAI.Testsuite is 203/203. Negative control: with the old sentence restored, 10 of 29 guard tests fail.Live status. 2 occurrences today, at ~08:31Z and ~10:43Z (
Ops/Actions/logs-20261008-a982-pii). Details are on #5932.Closing condition. #3179 is merged and served. Then either the next
redaction_context_lostline reads "refused this request under a content policy", or the preset is off and the line stops.rbuergi commentedon Oct 8, 2026
Systemorph/MeshWeaver.Plugins#3179 merged at 2026-10-08T18:17:53Z as
01c23c37bf0c. In review it gained three things:ProviderHealthRulecannot mark a key RateLimited on a plain 5xx.ProviderStreamException.RequestBlockedDataKey, so the thread cell renderschat.modelRequestBlockedonce the catalog has it. That key is i18n(chat): chat.modelRequestBlocked — the prose for a request the model gateway refused (#5960) #6325, open; until it lands, the guard's English stands.Closing condition, unchanged: the fix is served, and the next
redaction_context_lostreads as a content-policy block, or the preset is off and the line stops (#5932).Merge pull request #6325 from Systemorph/fix/5960-request-blocked-string
rbuergi commentedon Oct 9, 2026
#6325 merged at 2026-10-09T00:02:37Z as
2c98556648a7. It addschat.modelRequestBlockedin en and de. Mirror sync still owed: runnpm run sync:i18n -- --ref 2c98556648a7in MeshWeaver.Pluginsclients/react/src/i18n/. The key is additive and no value changed, so nothing is red meanwhile.systemorph commentedon Oct 9, 2026
Folded in — a DIFFERENT fingerprint from the same log site.
This issue tracks the log site
dc3faa884909b1cd(MeshWeaver.AI.AgentChatClient), and these occurrences belong to it. They carry their own fingerprint — the masking split one site across several — so before this they would have opened a SEPARATE ticket. One log site fanned out into 46 of those, which is why they fold here instead. Nothing is discarded: the distinguishing evidence is below, and splitting this back out is a person's call.8fe6a07af5b9dbecdc3faa884909b1cdMeshWeaver.AI.AgentChatClientMeshWeaver.AI.ProviderStreamExceptionMeshWeaver.AI.OpenAI.OpenAIWireStreamGuard.GetStreamingResponseAsync(IEnumerable1 messages, ChatOptions options, CancellationToken cancellationToken)+MoveNext()`The model provider answered with an error instead of a response: [{n}] Request blocked: PII detected (redaction_context_lost). The model gateway refused this request under a content policy (a guardrail such as PII redaction, or moderation). That is a verdict on the request's content, not a fault of the provider, its key or its account, and the same key goes on serving other requests.memexmemex-portal-deployment-5bb454569b-xfwpzAdmin/_LogIncident/8fe6a07af5b9dbecWhat triage wrote about this fingerprint
What is failing
The model gateway's PII redaction filter again rejects an outbound agent chat request with
403 Request blocked: PII detected (redaction_context_lost), failing closed and aborting the whole agent round mid-flight. In this occurrence the round that died was itself an automated log-incident triage round (Admin/_LogIncident/1ebb7190830184f0/_Thread/triage-…), so triage throughput is now also a casualty of the block.Probable cause
High confidence — this is the same mechanism already diagnosed for #5932: the gateway-side
PiiRedactionFiltercannot find/restore its per-request redaction context (TTL/eviction on long rounds was pinned in the #5932 diagnose record) and its fail-closed policy blocks the request instead of redacting statelessly. The 403 reason code names it directly.New signal in this sample — #5960's mislabel looks fixed, #5932's block does not. The
ProviderStreamExceptionmessage now reads "the model gateway refused this request under a content policy… a verdict on the request's content, not a fault of the provider, its key or its account" — the canned provider-account sentence that #5960 was filed against is gone, and the attribution is now correct. So the error-classification fix inOpenAIWireStreamGuardhas evidently landed, while the underlying fail-closed redaction block (#5932, stillbugPhase: blockedpending a human with access to the privateSystemorph/MeshWeaver.Pluginsrepo) remains unfixed and continues to abort rounds.Impact
One occurrence on one pod, but at least the third known aborted round from this defect (two pr-reviewer rounds in #5932/#5960, now one triage round). Each abort discards a completed analysis with nothing written back; workaround is to re-run the round. A secondary automated path — not release-blocking.
Known / duplicate status
Recurrence of #5932 (#5932 — fail-closed redaction context loss) and of #5960 (#5960, which stands for this log site
dc3faa884909b1cd; its attribution half appears fixed, this fingerprint is the fixed-message form of the same block). A human can close this ticket against #5932 in seconds — its value is the recurrence evidence and the confirmation that #5960's message fix shipped.Where to look
src/MeshWeaver.AI/OpenAI/PiiRedactionFilter.cs— theredaction_context_lostemission site; on a context miss, redact statelessly or re-establish context instead of blocking (per the PII redaction filter on the model gateway blocks agent chat requests outright when its redaction context is lost, aborting agent rounds mid-flight #5932 fix direction).src/MeshWeaver.AI.OpenAI/OpenAIWireStreamGuard.csGetStreamingResponseAsync(line 100 in the sample stack) — only for verification that the corrected attribution message is the shipped one.Still occurring: 1 more since the last update (1 total).
memex-portal-deployment-5bb454569b-xfwpzRecent log lines
systemorph commentedon Oct 9, 2026
Folded in — a DIFFERENT fingerprint from the same log site.
This issue tracks the log site
dc3faa884909b1cd(MeshWeaver.AI.AgentChatClient), and these occurrences belong to it. They carry their own fingerprint — the masking split one site across several — so before this they would have opened a SEPARATE ticket. One log site fanned out into 46 of those, which is why they fold here instead. Nothing is discarded: the distinguishing evidence is below, and splitting this back out is a person's call.b2b4b578910e5b22dc3faa884909b1cdMeshWeaver.AI.AgentChatClientMeshWeaver.AI.ProviderStreamExceptionMeshWeaver.AI.OpenAI.OpenAIWireStreamGuard.GetStreamingResponseAsync(IEnumerable1 messages, ChatOptions options, CancellationToken cancellationToken)+MoveNext()`The model provider answered with an error instead of a response: [{n}] This endpoint's maximum context length is {n} tokens. However, you requested about {n} tokens ({n} of text input, {n} of tool input). Please reduce the length of either one, or use the context-compression plugin to compress your prompt automatically.. The provider's reason is quoted above.memexmemex-portal-deployment-cfb58787b-p66hbAdmin/_LogIncident/b2b4b578910e5b22What triage wrote about this fingerprint
What is failing
A background agent thread's chat request exceeded the model endpoint's 1,048,576-token context limit (1,326,066 tokens requested, 1,320,951 of it text input) and the provider rejected it with HTTP 400. This is the same defect already filed as #5932 (incident
83e3b85b6f405c48): the chat client dispatches the full reconstructed conversation with no client-side context-length guard, truncation, or compression. The only change since #5932 is on the wire side — the OpenAI call is now wrapped byOpenAIWireStreamGuard, which converts the raw SDKClientResultExceptionintoProviderStreamExceptionbefore it reaches the[ThreadExec]error site. Treat this as a recurrence of #5932; a human can likely close this as a duplicate or re-open #5932 with the new stack.Probable cause — high confidence
The conversation history replayed into the round is larger than the endpoint's window. The prior incident's analysis holds: each round rebuilds the full conversation from the thread's stored messages and re-sends it, so a long-running thread with large content accumulates context monotonically until it crosses the limit. The concrete thread here is itself a log-triage agent (
Admin/_LogIncident/957a0437265f8931/_Thread/triage-the-log-incident-at-admin-loginci-a3de739091c09fd57050) — a triage thread working over an incident whose verbatim log samples are large, which is a natural way to blow a 1M-token budget. What still needs a decision is the framework fix: a pre-flight token estimate against the model node's declared limit plus automatic compression/truncation (the provider's error message itself points at the context-compression plugin, which the client is evidently not applying).Impact
The affected agent round dies loudly (no silent wrongness, no data loss) and the thread's task goes unprocessed for that round. Volume here is 1 occurrence on 1 pod; the systemic impact is that every long-running agent thread is exposed to the same wall, and a thread relaunched with its accumulated history will hit it again. This is a background automation path, so sev:M — same ruling as #5932.
Where to look
MeshWeaver.AI.OpenAI/OpenAIWireStreamGuard.GetStreamingResponseAsync(the exception site, stack line 100/124) — where the provider 400 is converted toProviderStreamException; note this wrapper is what changed since PII redaction filter on the model gateway blocks agent chat requests outright when its redaction context is lost, aborting agent rounds mid-flight #5932, changing the fingerprint.MeshWeaver.AI.AgentChatClient— request assembly; the pre-flight context check or automatic compression belongs here (per the fix direction already described in PII redaction filter on the model gateway blocks agent chat requests outright when its redaction context is lost, aborting agent rounds mid-flight #5932).MeshWeaver.AI/StreamStallGuardChatClient.Swallow(line 234) — where the exception is swallowed after logging; worth confirming the round terminates cleanly rather than retrying the identical over-limit request.Still occurring: 1 more since the last update (1 total).
memex-portal-deployment-cfb58787b-p66hbRecent log lines
rbuergi commentedon Oct 9, 2026
Verified on memex.systemorph.com. Closing: the guard half is live.
What is serving. Every memex replica runs
memex-portal-ai:3.0.0-ci.10310. That is 3/3 pods (cfb58787b-5d998,-b6klj,-p66hb), read with the governed Sample actionOps/Actions/sample-20261009-sevm-aiat 14:41Z. The image was built from core8b791643paired with Plugins7205855(main-cd run 37912557536, pair tag8b79164-p7205855). Plugins7205855contains #3179's merge01c23c37bf0c: GitHub compare reports01c23c37bf0c...7205855asahead,behind_by=0.The reading that decides it. The production line itself has changed. Incident fingerprint
8fe6a07af5b9dbec, folded here at 11:01Z today, is aredaction_context_lostblock at 2026-10-09 11:00:03Z. It now reads:On 2026-10-08 at 08:31Z and 10:43Z the same 403 still carried the old sentence, "This is a fault at the provider or the provider account …". So the old and new texts are seen on the same log site, on either side of the deploy.
What is not part of this issue and stays open:
b2b4b578910e5b22was folded into this issue at 11:30Z:[400] This endpoint's maximum context length is 1048576 tokens. However, you requested about 1326066 tokens, onAdmin/_LogIncident/957a0437265f8931/_Thread/…a3de739091c09fd57050. It is a different defect: an over-limit conversation is sent at all. It shares only the log site. I have not filed it separately, so triage may want to split it out.