Repository navigation
The Doc content gate CRASHES on a $model-probe hub-activation timeout — no verdict, on unrelated PRs #2468
Description
Two PRs whose diffs cannot reach the Doc content at all — #2467 (one workflow file) and #2461 (CI) — both went red on Doc content compiles and runs, with the same shape:
warn: MeshWeaver.Mesh.GetMeshNode[0] GetMeshNode('$model-probe/a75e76b9d2fa4ebbaca5e979f648ee2d') timed out
after 10.0s (budget 10s) — the owning per-node hub never answered the GetDataRequest.
Unhandled exception. System.AggregateException: One or more errors occurred.
---> System.TimeoutException: GetMeshNode('$model-probe/…') timed out after 10.0s …
at System.Reactive.PlatformServices.ExceptionServicesImpl.Rethrow(Exception)
…
at MeshWeaver.Mesh.MeshNodeStreamExtensions.<…>g__EmitError|2(Exception)
at System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean)
##[error]The content gate failed before it produced a verdict — no check was judged.
- CD: resolve the bake's image tag ONCE, and assert it matches why the run exists #2467 → job 98407873831, probe
a75e76b9… - Core builds the image and runs its OWN tests — plugins are built by the plugin repo #2461 → job 98355928593, probe
47eef047…
Same 10 s budget, different probe guid, unrelated diffs. #2456 / #2455 / #2451 passed the same check in the same window, so it is intermittent rather than a trunk break.
Two defects, and the second is the worse one
1. A $model-probe/<guid> per-node hub does not answer within 10 s. The message already says this is not "node not found" — the hub never answered its GetDataRequest. A transient probe address that never activates is a real bug or a budget that is too tight under CI load; either way it is diagnosable from the address family.
2. The timeout escapes as an UNHANDLED EXCEPTION and takes the process down. The gate then reports "failed before it produced a verdict — no check was judged" — so a run in which the gate learned nothing is presented in the same red as a run in which it judged content and found it wrong. Those are different facts and want different responses: the first is "re-run", the second is "fix your content". Worse, the failure is in the family this repo keeps re-learning — a check whose outcome does not mean what its colour says.
A Rethrow reaching ExceptionServicesImpl from a ReplaySubject.OnError means the probe path has no error handler at its subscription seam. The observable is doing the right thing (erroring); nobody is catching it.
What would close it
- The probe timeout should be an outcome the gate CLASSIFIES, not an exception that ends it — the same reasoning as "a gate that can never open must answer immediately rather than sit on a timeout".
- Distinguish "no verdict" from "verdict: fail" in the annotation, so a reader can tell a flake from a finding without opening the log.
- Then find out why a
$model-probehub misses a 10 s activation budget on a loaded runner.
Activity
rbuergi commentedon Aug 27, 2026
Escalation: this is now hitting
main, not only PRsRun 33085734292 on
mainat46fde128(15:14Z):Stack, ending in the same place as this issue's original report:
Also in the same job, and possibly a second defect:
Hub sync/… initialization failed — a BuildupAction …(compare #2444), plus aReplaySubject.ReplayBase.OnErrorframe.The commits under it are three merges I made minutes earlier (#2470, #2475, #2484). None of their diffs touches
Doc/content orMeshNodeStreamExtensions— #2470 is a workflow + AGENTS.md, #2475 is the combo-verifier and its own tests, #2484 is the recursive-delete fan-out. So this is the known defect firing on main rather than a regression from those merges; recording the attribution here rather than leaving the red unexplained.Why the escalation matters beyond one red run
A gate that dies before judging is worse than a gate that fails. "No check was judged" and "every check passed" are one glance apart on a dashboard, and this repo's standing rule is that a gate which cannot report is not a gate. On
mainspecifically, the required check going red for a reason unrelated to the tree blocks the merge queue for everyone and trains people to discount it — which is precisely how a real content failure would get waved through.The mechanism, stated for whoever picks it up
A
GetMeshNodeOutcometimeout raisesobserver.OnError(...)from acts.Token.Registercallback on aTimerQueuethread, into a chain terminating at aSubscribe(onNext)with no error arm. Rx's default handler rethrows on that thread, which is not a test failure but an unhandled exception that kills the host — hence no verdict.Two things need separating, and I have not established which is in play here:
the owning per-node hub never answered the GetDataRequest. If a$model-probehub never answers, that is its own defect. Prior art worth checking:5ddaf0d4b"A transient probe hub never starts its data sources — the CD-killing flake" anda496723b4splittingDataContext.Initialize, both 2026-08-22 — this may be a survivor of that family.🚨 Not to be fixed by widening the 10 s budget. If the hub never answers, the bound is not the problem.
An agent is on this now.