Skip to content

fix(#3625): the untyped-content gate can fire, and keys on the event not the sentence - #3628

Merged
meshweaver-cloud[bot] merged 7 commits into
mainfrom
fix/3625-the-untyped-content-gate-can-fire
Sep 7, 2026
Merged

meshweaver-cloud[bot] merged 7 commits into
mainfrom
fix/3625-the-untyped-content-gate-can-fire

Conversation

@rbuergi

@rbuergi rbuergi commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

Closes #3625.

The gate could not fire, and that is worse than a gate that is merely wrong

dotnet-test.yml runs check-untyped-content.sh collected-logs on every shard. The script is
careful in every way that matters given its input: it separates "found nothing" from "the scan
failed" on grep's exit code, refuses a missing directory, and explains the fix. Its input could
never arrive.

Both producers logged the degradation as a Warning with no exception:

  • MeshNodeStreamCache.cs — GetStream's converter
  • MeshNodeStreamCache.cs — GetQuery's converter

and the sink that writes collected-logs/_meshweaver-test-trace.log takes a record if and only
if
exception is not null && logLevel >= LogLevel.Warning
(XUnitFileLogger.Log / LoggingBuilderExtensions.Log → TestTraceLog.AppendFault). The other
route into that directory — per-method */bin/*/test-logs/*.log — is opt-in via
MESHWEAVER_TEST_FILE_LOGS, which node-repo-module-pack.yml sets and dotnet-test.yml does not.
So the phrase reached the scanned directory by construction never: permanently green, and
therefore indistinguishable from a check that passed. That is the one reading AGENTS.md singles out
as forbidden, committed inside a gate written to prevent it.

The existing control test asked the wrong question. UntypedContentDegradationGate pins that
the source still emits the phrase and that the script still greps it. Both were true the whole
time. Neither asks whether a record can reach the directory the script scans — a guard whose
subject moved and whose roots did not passes having checked nothing.

This is the third recorded instance of the same trap: #890's PROCESS CANNOT EMIT and #3413's
never rendered each lost their own explanation to it. The new part is that this time it disarmed a
gate.

The fix — and why the gate no longer keys on prose

1. The record can reach the sink. The three degradation sites now pass
MeshNodeContentDegradedException — an exception object constructed and never thrown, whose
entire job is to satisfy the sink's predicate. Reaching a sink is a property of the CALL, not of the
level or the wording; this is the same fix #890 got, one diagnostic over. It captures no stack (it
is never thrown), so ToString() is exactly type: message — which is the whole record a reader
needs, because the fault is in the DATA, not in a call path.

2. The gate keys on the TYPE, not the sentence. A message is a description of the event; the
type is the event's identity, bound by the compiler at every construction site. The phrase had
exactly one binding — this test file. The type now has two: renaming it is a repo-wide compile
change, and UntypedContentDegradationGate pins the script's MARKER= to
nameof(MeshNodeContentDegradedException). The phrase is kept as a second net because it costs
nothing and a per-test file log carries the formatted message with no exception attached at all.

3. A third blind spot, closed on the way. GetQuery's deserialization catch branch already
carried an exception and so already reached the trace — but its wording is "stays", not
"stayed", so the phrase grep could never have matched it either. Keying on the type covers it;
ex rides along as the inner exception, so the deserialization stack is still in the record.

4. The missing control. UntypedContentDegradationReachesTheTraceSinkTest drives the
production converter (both seams, made internal for exactly this) and evaluates the sink's own
condition against the captured record — pinning the sink predicate, not the call, because asserting
"we passed an exception" would restate the fix in the fix's own terms. Modelled on
EmitDeadAttributionReachesTheTraceSinkTest, which the maintainer named as the template.

It captures through its own recording logger and never touches TestTraceLog. That is deliberate,
and it is the objection that kept this control from being written before: a control arm that wrote a
real degradation into the shared trace file would red every shard. The chain is split cleanly —
emitter → record is exercised by the test, record → file is AppendFault's own contract, and
file → verdict is the script, falsified directly below.

No mocking. The discriminating variable is a real MeshContentTypeRegistry: empty in the
degraded arm, carrying the probe type in the non-vacuity arm. Same node, same converter, same
options — the only difference is whether the mesh knows the type, which is exactly the condition the
warning reports.

🚨 Falsification — the gate made to go red, then green, exit codes read directly

The same degradation, through the same real sink (XUnitFileLogger → TestTraceLog.AppendFault),
under a redirected TMPDIR so nothing touched the shared trace file. Three runs:

arm what was in collected-logs/ gate
fix reverted (exception argument removed at the GetStream seam) the trace file was never created No content-type degradation — exit 0
fix in place the [FAULT] record, carrying the phrase and MeshWeaver.Mesh.MeshNodeContentDegradedException exit 1, 2 Space/ARenderedEmptyPage
control: a real but unrelated [FAULT] (InvalidOperationException, 223 bytes) a non-empty, genuine trace with no degradation exit 0

The first row is the defect reproduced: the condition the gate exists to catch, occurring, and the
gate reporting success. The third row is what makes the second row a verdict rather than an empty
scan — the gate deliberately passes on an empty directory, so a green arm with nothing in it would
prove nothing.

The harness that produced rows 1 and 2 is deliberately not committed: in CI it would write a
genuine degradation into the shared trace file and red every shard — which is the gate working
correctly.

Verification

  • dotnet build -c Release -warnaserror, one project per invocation, exit read directly (never
    through a pipe): MeshWeaver.Hosting, MeshWeaver.Hosting.Test, MeshWeaver.Documentation.Test
    — all 0 Warning(s) 0 Error(s).
  • UntypedContentDegradationReachesTheTraceSinkTest 4/4 (2 seams × degraded + recovered).
  • UntypedContentDegradationGate 4/4 (2 pre-existing + 2 new), plus
    DocumentationLinkIntegrityTest and WhatsNewEntryIntegrityTest green after the doc edits — 6/6
    in that filter.

Work product conserved

Doc/Architecture/ReadingCiSignals → the fault-sink section now carries this as the blind spot's
third instance, with the measurement, why the control test missed it, and the falsification table.
That section already documented instances one and two.

Risk, stated plainly

A test that deliberately seeds unreadable node content will now red its shard, because its
degradation finally reaches the sink. That is the gate doing its job — but it is a behaviour change
to CI, not a refactor, and if a shard reds on this after merge the right response is to fix the
degradation (register the content type where it is READ), never to widen the gate.

What's New: Category: Fix — "A content type nobody registered can no longer slip through CI".

🤖 Generated with Claude Code

…not the sentence

`check-untyped-content.sh` greps a shard's `collected-logs/` for a content degradation
and reds on a hit. It could not fire: both producers logged the phrase as a Warning with
NO exception, and the sink that writes `_meshweaver-test-trace.log` takes a record if and
only if `exception is not null && logLevel >= Warning`. The phrase could reach the
directory the script scans by construction never — permanently green, and therefore
indistinguishable from a check that passed.

The two `LogWarning` calls in `MeshNodeStreamCache` (and `GetQuery`'s catch branch, whose
wording "stays" the phrase grep never matched anyway) now carry
`MeshNodeContentDegradedException` — constructed, never thrown, its whole job being to
satisfy the sink's predicate. The gate keys on that TYPE NAME first and on the prose only
as a second net: a message describes an event, a type IS the event, bound by the compiler
at every construction site, so the coupling gains a binding it did not have.

`UntypedContentDegradationReachesTheTraceSinkTest` supplies the half the existing control
never asked: it drives the production converter and evaluates the sink's own condition
against the captured record, with a non-vacuity arm on the same node through a registry
that CAN recover it. `UntypedContentDegradationGate` now also pins the script's key to
`nameof(MeshNodeContentDegradedException)` and requires every seam to attach it.

Falsified end to end, exit codes read directly: fix reverted -> no trace record written,
gate exit 0; fix in place -> record written, gate exit 1; unrelated real fault only ->
gate exit 0.

Refs #3625

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Copilot AI lite review requested due to automatic review settings September 7, 2026 16:23
…and the shape that replaces one

A test proving a write REFUSES unreadable content must seed unreadable content, and once
these records reach the sink such a fixture reds its own shard. An exemption list would be
a permanently green check wearing a reason. The script and the doc now name the shape that
removes the pressure: seed a value of a DIFFERENT, REGISTERED type — the cache types it, so
nothing degrades and nothing is recorded, while ContentAs<T> still answers null because it
recovers a foreign runtime type only when the short name matches. That is the production
case (a same-named record from another collectible assembly), so the fixture gets stronger;
malformed JSON with no resolvable $type models the OTHER defect, the one the gate reports.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The updated gate script currently greps all files recursively without excluding binary artifacts in collected-logs/, which can cause false positives and misleading output (and the control test has a brittle construction-count heuristic).

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR fixes the CI “untyped content degradation” gate so it can actually trigger by ensuring the degradation diagnostics reach the shard trace sink, and makes the gate more robust by keying on an exception type name (event identity) in addition to the prose phrase. It also adds a reachability control test and documents the blind spot and its resolution.

Changes:

  • Attach a purpose-built (never-thrown) MeshNodeContentDegradedException to degradation warnings so they satisfy the trace sink’s admission predicate.
  • Update check-untyped-content.sh and its control test to key primarily on the exception type name (with the phrase retained as a secondary net).
  • Add a production-path reachability test plus supporting documentation/What’s New entry.
File summaries
File Description
src/MeshWeaver.Hosting/MeshNodeStreamCache.cs Attaches the marker exception at all degradation seams and exposes internal seams for testing.
src/MeshWeaver.Data.Contract/MeshNodeContentDegradedException.cs Introduces the marker exception type used as the gate’s identity key.
.github/scripts/check-untyped-content.sh Updates the gate to scan for marker type name and phrase, and improves reporting.
test/MeshWeaver.Documentation.Test/UntypedContentDegradationGate.cs Extends the control test to pin the script’s marker key and enforce seam coverage.
test/MeshWeaver.Hosting.Test/UntypedContentDegradationReachesTheTraceSinkTest.cs Adds a production-path reachability control that evaluates the sink predicate on captured records.
src/MeshWeaver.Documentation/Data/Architecture/ReadingCiSignals.md Documents the third instance of the sink blind spot and the new controls.
src/MeshWeaver.Documentation/Data/WhatsNew/2026-09-07-a-content-type-nobody-registered-can-no-longer-slip-through-ci.md Adds a user-facing “Fix” entry describing the CI behavior change.
Review details
  • Files reviewed: 7/7 changed files
  • Comments generated: 4
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment on lines 91 to 93
set +e
matches=$(grep -rl "$PHRASE" "$DIR" 2>"$scan_err")
matches=$(grep -rl -e "$MARKER" -e "$PHRASE" "$DIR" 2>"$scan_err")
scan_rc=$?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adopted, with a different instrument — and applied to both greps in one shared EXCLUDES array so they cannot drift.

The concern is real: dotnet-test.yml copies /tmp/coredumps/*.dmp into collected-logs/, plus a symbols-<project>/ directory of .dll + .pdb whenever a host dies on a signal. A core dump is a snapshot of process memory, so it can contain the marker simply because the process once formatted it — a match there is not a log record.

I did not use -I, and the reason is the same rule this whole script exists to enforce. -I decides by CONTENT: a text log carrying one stray NUL byte becomes invisible to the scan, silently, and the gate then reports "No content-type degradation" — a pass-by-scanning-nothing, which is the one outcome this file must make impossible. A false red is loud and investigable; a false green is not. Excluding by NAME (--exclude='*.dmp' --exclude='*.dll' --exclude='*.pdb') can only ever skip an extension listed in the open, right there in the script, and the reasoning is written beside it.

Comment on lines +115 to 120
# Both record shapes name the node: the log message says "Content for <path> stayed …", the
# exception says "content for '<path>' (nodeType …)". Reduce either to the path.
grep -rh -e "$MARKER" -e "$PHRASE" "$DIR" 2>/dev/null \
| sed -E -e "s/.*Content for ([^ ]+) stayed.*/ \1/" \
-e "s/.*content for '([^']*)'.*/ \1/" \
| sort | uniq -c | sort -rn | head -20

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes — that was the second half of it, and the fix covers both call sites: the EXCLUDES array is applied to the grep -rl scan and to the grep -rh Occurrences pass, so a binary artefact can neither red the gate nor emit a Binary file … matches line that the sed path-extraction cannot reduce.

(Name-based rather than -I — see the reply on the scan line for why deciding by content is the wrong direction in this particular script.)

Comment on lines +87 to +89
var source = Read(EmittingSource);
var constructions = source.Split('\n').Count(l => l.Contains("new " + Marker, StringComparison.Ordinal));

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed, and taken one step further than suggested, because a regex alone would have left the second half of your point live.

MeshNodeStreamCache.cs is a file that explains itself at length, and its comments name MeshNodeContentDegradedException repeatedly — so "fooled by a comment" is the normal case here, not the edge case. The count now runs over SourceScan.MaskCommentsAndStrings(...) first (the helper every other source ratchet in this repo uses, which blanks comments and string literals while preserving offsets), and then matches:

\bnew\s+(?:[A-Za-z_][\w]*\s*\.\s*)*MeshNodeContentDegradedException\s*\(

which also handles the wrapped-across-lines and alias-qualified forms you named. Verified green in the full MeshWeaver.Documentation.Test suite (325/325, Release).

Comment on lines +170 to +173
public IDisposable BeginScope<TState>(TState state) where TState : notnull => NullScope.Instance;

public bool IsEnabled(LogLevel logLevel) => true;

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adopted — plus the change that actually fixes the brittleness you describe, because narrowing IsEnabled on its own would have been a no-op here.

LoggerExtensions.LogWarning(...) and friends call ILogger.Log(...) unconditionally; they do not consult IsEnabled. This recorder's Log recorded everything it was handed, so IsEnabled was never on the path under test — an added Info/Debug line would still have broken Assert.Single.

What makes the control robust is filtering the asserted population by the sink's own predicate:

  • Assert.Single(logger.Records.Where(ReachesTheTraceSink))
  • Assert.Empty(logger.Records.Where(ReachesTheTraceSink))

A future Info/Debug record — which the sink drops anyway, so it says nothing about reachability — can no longer fail these; a second warning-with-exception still does, because two records reaching the sink is a real change to what CI sees.

Your suggestion is in as the other half: IsEnabled is now >= LogLevel.Warning and Log honours it, so the recorder is a faithful logger rather than a tee, and what it holds is what the sink would consider.

@github-actions

github-actions Bot commented Sep 7, 2026 •

Copy link
Copy Markdown
Contributor

Test Results (shard 0)

251 tests  ±0   251 ✅ ±0   2m 6s ⏱️ +23s
  1 suites ±0     0 💤 ±0 
  1 files   ±0     0 ❌ ±0 

Results for commit 3f232aa. ± Comparison against base commit da6f49a.

♻️ This comment has been updated with latest results.

@github-actions

github-actions Bot commented Sep 7, 2026 •

Copy link
Copy Markdown
Contributor

Test Results (shard 3)

1 061 tests  ±0   1 061 ✅ ±0   1m 16s ⏱️ -12s
    2 suites ±0       0 💤 ±0 
    2 files   ±0       0 ❌ ±0 

Results for commit 3f232aa. ± Comparison against base commit da6f49a.

♻️ This comment has been updated with latest results.

@github-actions

github-actions Bot commented Sep 7, 2026 •

Copy link
Copy Markdown
Contributor

Test Results (shard 1)

492 tests  ±0   492 ✅ ±0   38s ⏱️ -3s
  1 suites ±0     0 💤 ±0 
  1 files   ±0     0 ❌ ±0 

Results for commit 3f232aa. ± Comparison against base commit da6f49a.

♻️ This comment has been updated with latest results.

@github-actions

github-actions Bot commented Sep 7, 2026 •

Copy link
Copy Markdown
Contributor

Test Results (shard 4)

1 666 tests  +14   1 472 ✅ +14   2m 10s ⏱️ +20s
    4 suites ± 0     194 💤 ± 0 
    4 files   ± 0       0 ❌ ± 0 

Results for commit 3f232aa. ± Comparison against base commit da6f49a.

♻️ This comment has been updated with latest results.

@github-actions

github-actions Bot commented Sep 7, 2026 •

Copy link
Copy Markdown
Contributor

Test Results (shard 5)

    5 files  ±0      5 suites  ±0   2m 35s ⏱️ -3s
1 586 tests ±0  1 586 ✅ ±0  0 💤 ±0  0 ❌ ±0 
1 587 runs  ±0  1 587 ✅ ±0  0 💤 ±0  0 ❌ ±0 

Results for commit 3f232aa. ± Comparison against base commit da6f49a.

♻️ This comment has been updated with latest results.

@github-actions

github-actions Bot commented Sep 7, 2026 •

Copy link
Copy Markdown
Contributor

Test Results (shard 2)

1 156 tests  +10   1 156 ✅ +10   4m 26s ⏱️ +8s
    3 suites ± 0       0 💤 ± 0 
    3 files   ± 0       0 ❌ ± 0 

Results for commit 3f232aa. ± Comparison against base commit da6f49a.

♻️ This comment has been updated with latest results.

@github-actions

github-actions Bot commented Sep 7, 2026 •

Copy link
Copy Markdown
Contributor

Test Results

   16 files  ± 0     16 suites  ±0   13m 14s ⏱️ +34s
6 212 tests +24  6 018 ✅ +24  194 💤 ±0  0 ❌ ±0 
6 213 runs  +24  6 019 ✅ +24  194 💤 ±0  0 ❌ ±0 

Results for commit 3f232aa. ± Comparison against base commit da6f49a.

♻️ This comment has been updated with latest results.

rbuergi and others added 3 commits September 7, 2026 18:43
…e that models production

Measured, not predicted: with the gate able to fire, #3619's UnreadablePolicyRecordIsNotClobbered
put EIGHT MeshNodeContentDegradedException records into the trace log and would have red its shard
on every run — a TRUE positive, because its seeded JSON carries no resolvable $type and is
therefore content nothing can read, which is precisely what the gate reports.

An allow-list was refused: it would be a permanently green check wearing a reason. The fixture is
what changes, and the change makes it stronger. The record now names a REGISTERED foreign type
(ForeignPolicyRecord — the same fields, with comboVerifications as a string where this build
declares a list), so the stream cache materialises it and nothing degrades, while
As<UpdatePolicyContent> still answers null because it recovers a foreign runtime type only when
the short name matches. That is the production case the fixture's own comment already claimed to
model — "a field written by a build whose content type differed" — and it was not modelling it.

Falsified both directions, against the SAME fixture: reverting #3619's two src files fails exactly
the 4 refusal arms with the 4 positive controls passing (#3619's own reported result, unchanged);
with the fix, 8/8, and the marker count in the trace goes 8 -> 0 with the gate exiting 0.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…entory was unreadable

With the gate able to fire, its first working CI run reported Ops/Modules/unit-dep degrading
on shard 5. It is a TRUE POSITIVE, and a live one in src/.

DeploymentReportService.WriteLocally stamps a $type onto the inventory document, and the
constant it stamped was the literal "ModuleInventoryContent" — a name NO CLR type in the
fleet carries. Its own doc comment says why the stamp exists ("content without the
discriminator is stored perfectly and materialises as NOTHING") and the stamp did not
achieve it: the reading hub could not resolve the name, ObjectPolymorphicConverter degraded
the value back to a raw JsonElement, and the node materialised as nothing anyway. The real
record, DeploymentReport, was registered nowhere — absent from AddPluginCatalogTypes, which
is applied to both the mesh hub and the default node hub.

So every instance's self-reported module inventory at Ops/Modules/{deployment} read back
untyped, and every reader doing `Content is DeploymentReport` saw an empty record. Nothing
could see it: the write succeeds, the row is perfect, and only the readers come up empty.

Both halves fixed — DeploymentReport is registered, and InventoryContentType is now
nameof(DeploymentReport) rather than a hand-typed string, so the two cannot drift again.

🚨 The guard test asserted the DEFECT. AnInstanceReportsWhatItRunsTest checked
`content.GetProperty("$type") == InventoryContentType` — and a `$type` PROPERTY is only
there to be read when the content is raw JSON, i.e. when it has NOT materialised. The
assertion passed BECAUSE of the bug. It now asserts the materialised runtime type, which is
what every reader actually depends on.

Not ContentAs<T> either: that is the bad-data-TOLERANT accessor and deserialises a raw
JsonElement regardless of the discriminator, so it answers a DeploymentReport either way —
an assertion that cannot fail. Measured, not assumed: the ContentAs version passed against
the reverted fix.

Falsified: reverting the discriminator alone fails the new assertion with "Expected value to
be of type DeploymentReport ... but found JsonElement". With the fix, 976/976 in
Memex.Portal.Shared.Test and the degradation marker count on that test goes 2 -> 0.

Refs #3625

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… trap it exposed

Two occurrences on the first working run, opposite verdicts: Ops/Modules/{deployment} is a
TRUE POSITIVE (a discriminator naming no CLR type, the record registered nowhere), and
LateContentTypeRegistrationTest is a true degradation that is the test's own SUBJECT — the
keying is right, the gate simply cannot tell intent.

Also records the trap that cost a cycle: the guard on the true positive asserted the DEFECT
(a $type PROPERTY is only readable while the content has NOT materialised), and the obvious
replacement — ContentAs<T> — could not fail either, because it is the bad-data TOLERANT
accessor. Measured, not assumed. Assert the runtime TYPE when a diagnostic says a value
reads as absent.

Leaves the open policy question stated plainly: how a test declares a degradation as its
subject, when the registered-foreign-type fixture shape cannot apply.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…— no allow-list, nothing skipped

The repaired gate red shard 4 on LateContentTypeRegistrationTest. Measured on run 34150643931, and
the denominator is smaller than it looked: the Occurrences list is exactly lcu…/Live, lcr…/Live,
lcn…/Live — that one class's three tests. The OverviewUnresolvedContentTypeTest warnings in the same
shard come from ObjectPolymorphicConverter, carry no marker and say "can only be read as", so they
never matched the gate at all.

Neither answer the gate offers applies here.

  * An ALLOW-LIST is refused on principle — and these paths carry a per-run GUID, so the entry
    would have to be a PATTERN: an exemption that widens by itself.
  * The prescribed FIXTURE CHANGE (seed a DIFFERENT, REGISTERED type — f4b0420's repair of
    #3619's fixture) deletes the subject here. #2952 is about a NodeType compiled at RUNTIME into a
    collectible assembly, so "registered nowhere yet" IS the premise, and
    AnUnrelatedRegistration_LeavesGenuinelyUnresolvableContentUntyped asserts that content nothing
    can resolve STAYS untyped.

So the third answer, and it is the pattern this PR already set one test over.
UntypedContentDegradationReachesTheTraceSinkTest drives the production emitter against its OWN
recording logger. The emitter takes its logger from DI, so the same move works for a whole mesh:
LateContentTypeRegistrationTest substitutes the CLOSED ILogger<MeshNodeStreamCache> for its own mesh
with a decorator that takes MeshNodeContentDegradedException records and forwards everything else
untouched.

🚨 Why that is not the skip-trapdoor an allow-list would be — four properties, all four needed:

  1. Nothing is silenced. No level is dialled, no category is muted; every other record the cache
     logs reaches the real logger at its own level.
  2. THE DIVERSION IS AN ASSERTION. Each test asserts the record WAS produced, that it names the
     expected node, and that it satisfies the trace sink's OWN predicate
     (exception is not null && level >= Warning). A declaration that matches nothing is a RED —
     the one property an allow-list can never have, and strictly more than these tests asserted
     before, when the record went to a file nobody read.
  3. The exemption is exactly ONE NODE wide: the same assertion pins that everything captured names
     the declared path, so an unintended degradation elsewhere in that mesh fails the test rather
     than being swallowed.
  4. UntypedContentDegradationGate.ADivertedDegradationIsAssertedWhereItIsDiverted is the control
     arm, with a denominator: it reds if any test diverts without asserting, AND if it ever
     examines zero files.

FALSIFIED, not assumed, in both directions:

  * exception argument reverted at the GetStream seam (the #3625 defect itself) -> all 3 tests FAIL
    on "the platform must REPORT the degradation this test reproduces". The regression is now
    caught by a TEST, not only by a gate that must first see a whole shard's logs.
  * with the fix -> 3/3 pass, marker count in the shared trace 3 -> 0,
    check-untyped-content.sh exit 1 -> exit 0.
  * the new guard's marker mutated -> it reds naming LateContentTypeRegistrationTest.cs, so it is
    reading the production tree and the check bites.
  * Release + -warnaserror, one project per invocation: MeshWeaver.Documentation,
    MeshWeaver.Documentation.Test, MeshWeaver.Graph.Test, MeshWeaver.Hosting.Test — 0 Warning(s),
    0 Error(s) each.

🚨 NOT fixed here, filed as #3645: the gate cannot tell a TRANSIENT degradation from a FINAL one.
MeshNodeStreamCache warns at the instant of a read; MeshNodeTypeSource logs the same event naming
both causes ("(a) … YET, which is TRANSIENT … (b) no declaration will ever claim this") and carries
no exception, so it reds nothing. One event, two descriptions, opposite CI consequences — and two of
these three tests degrade AND RECOVER inside one test. Telling them apart needs a window nothing in
a process naturally closes: a rework, not this PR's red.

Review (Copilot x4), all four addressed:

  * check-untyped-content.sh now excludes the binary artefacts by NAME on BOTH greps
    (--exclude='*.dmp' '*.dll' '*.pdb'; collected-logs/ carries mini dumps and a symbols-<proj>/
    of dll+pdb). Deliberately NOT `-I`: that decides by CONTENT, so one stray NUL in a text log
    makes it invisible and the gate reports "no degradation" — a pass-by-scanning-nothing, which
    is the outcome this script exists to make impossible. A false red is loud; a false green is not.
  * the seam count is a regex over comment/string-MASKED source, so a prose mention of the type in
    that heavily-commented file cannot be counted and a wrapped or alias-qualified construction
    cannot be missed.
  * the reachability control asserts over the SINK-PREDICATE population rather than every record;
    RecordingLogger.IsEnabled is Warning+ and Log honours it. (Narrowing IsEnabled alone would have
    been a no-op — LoggerExtensions call ILogger.Log unconditionally.)

Doc/Architecture/ReadingCiSignals now answers the open policy question it had left standing, and the
What's New entry carries the user-facing half plus the transient caveat.

Refs #3625, #3645

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CCM41UknXQwgXqH39tLo9u
@rbuergi

rbuergi commented Sep 7, 2026

Copy link
Copy Markdown
Contributor Author

The shard-4 red, resolved — and why it is not an exemption

What actually red the gate. Run 34150643931, step "Gate: no content degraded to an untyped JsonElement". The Occurrences list is exactly three paths — lcu…/Live, lcr…/Live, lcn…/Live — i.e. LateContentTypeRegistrationTest's three tests, and nothing else in core. (The OverviewUnresolvedContentTypeTest warnings in the same shard come from ObjectPolymorphicConverter, carry no marker exception and say "can only be read as", so they never matched either key.)

Neither of the gate's two answers applies. An allow-list is refused on principle — and these paths carry a per-run GUID, so the entry would have to be a pattern: an exemption that widens by itself. The prescribed fixture change (seed a DIFFERENT, REGISTERED type — f4b0420e4's repair of #3619) deletes the subject here: #2952 is about a NodeType compiled at RUNTIME into a collectible assembly, so "registered nowhere yet" IS the premise, and AnUnrelatedRegistration_LeavesGenuinelyUnresolvableContentUntyped asserts that content nothing can resolve stays untyped.

So: the pattern this PR already set one test over. UntypedContentDegradationReachesTheTraceSinkTest drives the production emitter against its OWN recording logger. The emitter takes its logger from DI, so the same move works for a whole mesh — LateContentTypeRegistrationTest substitutes the CLOSED ILogger<MeshNodeStreamCache> for its own mesh with a decorator that takes MeshNodeContentDegradedException records and forwards everything else untouched.

Why that is not a skip-trapdoor

  1. Nothing is silenced. No level dialled, no category muted; every other record the cache logs reaches the real logger at its own level.
  2. 🚨 The diversion IS an assertion. Each test asserts the record was produced, that it names the expected node, and that it satisfies the trace sink's own predicate (exception is not null && level >= Warning). A declaration that matches nothing is a RED — the one property an allow-list can never have, and strictly more than these tests asserted before, when the record went to a file nobody read.
  3. The exemption is exactly ONE NODE wide. The same assertion pins that everything captured names the declared path, so an unintended degradation elsewhere in that mesh fails the test rather than being swallowed.
  4. UntypedContentDegradationGate.ADivertedDegradationIsAssertedWhereItIsDiverted is the control arm, with a denominator: it reds if any test diverts without asserting, and if it ever examines zero files.

Falsified in both directions, locally

experiment result
exception argument reverted at the GetStream seam (the #3625 defect itself) all 3 tests FAIL on "the platform must REPORT the degradation this test reproduces" — so the regression is now caught by a TEST, not only by a gate that must first see a whole shard's logs
with the fix 3/3 pass; marker count in the shared trace 3 → 0; check-untyped-content.sh exit 1 → exit 0
new guard's marker mutated reds, naming LateContentTypeRegistrationTest.cs — it is reading the production tree and the check bites
MeshWeaver.Documentation.Test full suite, Release 325 / 325
Release + -warnaserror, one project per invocation (MeshWeaver.Documentation, .Documentation.Test, .Graph.Test, .Hosting.Test) 0 Warning(s), 0 Error(s) each

The transient question — answered, and deliberately NOT fixed here

MeshNodeStreamCache warns at the instant of a read and cannot know the type will register moments later — the normal state during portal boot, and exactly the race #2952 fixed. The sibling seam already says so: MeshNodeTypeSource prints both causes ("(a) … YET, which is TRANSIENT … (b) no declaration will ever claim this") and carries no exception, so it reds nothing. One event, two descriptions, opposite CI consequences — and two of these three tests degrade and recover inside a single test. Telling them apart needs a window nothing in a process naturally closes: a rework, not this PR's red. Filed as #3645.

Review

All four Copilot comments addressed inline (binary exclusion by NAME rather than -I, and why; masked-source regex for the seam count; sink-predicate population for the reachability control). Reasoning conserved in Doc/Architecture/ReadingCiSignals — which now answers the open policy question it had left standing — and in the What's New entry.

@meshweaver-cloud
meshweaver-cloud Bot added this pull request to the merge queue Sep 7, 2026
Merged via the queue into main with commit a57a6b5 Sep 7, 2026
30 checks passed
rbuergi added a commit that referenced this pull request Sep 9, 2026
A node whose content carries an unresolvable $type degrades to a raw
JsonElement and reads as absent everywhere downstream — no exception, no NACK.
That silence is why there is a gate. It was catching the wrong population.

check-untyped-content.sh keyed on MeshNodeContentDegradedException, which is
emitted at the INSTANT of a read, and at that instant nothing can know whether
the type registers a moment later. That is the ordinary boot race: a NodeType's
runtime compile lands after its first readers are served, and #2952 exists to
re-type those readers. The seam one layer down already said so — "stays an
untyped JsonElement FOR NOW … (a) has not registered it YET, which is TRANSIENT
… or (b) no declaration will ever claim this discriminator" — and, passing no
exception, reddened nothing. One event, two descriptions, opposite CI
consequences. Measured on #3628: two of LateContentTypeRegistrationTest's three
cases degrade and RECOVER inside a single test, and both reddened a shard.

Nothing new is invented. ContentDegradationRegistry already kept one entry per
node type (added for /health), and IMeshContentTypeRegistry already answers "is
this resolvable" as a pure map lookup by either route TryRecoverForNodeType
takes. So the verdict is computed by ASKING AGAIN:

  * Record now keeps the stored $type alongside the NodeType, so both routes can
    be re-asked later without retaining the document;
  * Unresolved(registry) drops every entry whose type has since registered;
  * MeshNodeStreamCache.Dispose logs one MeshNodeContentUnresolvedException per
    surviving entry — first, before any teardown, so the verdict is always
    emitted;
  * the script keys on that type. The per-read records stay in the logs and stay
    useful: once the verdict names a type, they say which reads degraded on it.

/health was wrong the same way: ContentTypeHealthCheck fed Snapshot(), so a
replica stayed Degraded after a boot race until some later read happened to
clear the entry — a probe answering about the past. It re-asks now.

Falsified in both directions, as the issue asked: reduce Unresolved to
Snapshot() and ARecoveredDegradation_IsNotTheVerdict fails; remove the teardown
report and AnUnrecoverableDegradation_IsTheVerdict_AndReachesTheSink fails on
"the teardown must REPORT what never resolved". UntypedContentDegradationGate
additionally pins that the script keys on the verdict type, that the emitter
constructs it, that the report is computed from Unresolved(, and that both read
seams still emit the record the verdict is derived from.

🚨 Both messages keep the gate's phrase contiguous in ONE string literal: the
guard greps the source line by line, so a phrase the formatter splits across two
literals is emitted correctly and reads as absent to the guard — a rewording
that "passes" while retiring the coupling. Caught by the guard itself.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@rbuergi
rbuergi deleted the fix/3625-the-untyped-content-gate-can-fire branch September 13, 2026 10:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants