Network-dependent tests skip loudly on feed outage; watchdog measures monotonic time (#679) by rbuergi · Pull Request #885 · Systemorph/MeshWeaver

rbuergi · 2026-08-07T06:22:13Z

Summary

Fixes #679: when nuget.org has a bad minute, NuGetAssemblyResolverTest (3 tests) and MemexTemplateGeneratorTest (7 tests) went red with failures naming NuGet internals — phantom flakes an engineer then triages. And the MonolithMeshTestBase watchdog measured wall-clock time, so host suspension (laptop sleep / runner stall) produced impossible 5000+s "hangs" against a 720 s cap, all green on re-run under caffeinate.

Changes

Feed reachability → loud skip, never phantom red

Each NuGetAssemblyResolverTest test probes api.nuget.org/v3/index.json (10 s budget) and Assert.Skips with the cause when unreachable/unhealthy. The MESHWEAVER_SKIP_NUGET=1 opt-out was a silent return (vacuous pass) — now an explicit skip.
MemexTemplateGeneratorTest.RunGenerator gets the same probe (the network edge is dotnet run's implicit restore of the generator script), plus a detector for the probe-passed-then-feed-died window: a non-zero exit whose output names NU1900 / "unable to load the service index" skips as an outage; any other failure still fails as a generator defect.

Watchdog: monotonic enforcement + suspension report

Deadlines are enforced on a Stopwatch (does not advance across suspension on Linux CLOCK_MONOTONIC / Intel macOS) instead of DateTimeOffset.UtcNow deltas.
Wall-vs-monotonic divergence > 10 s logs [WATCHDOG-SUSPEND] naming the suspension window. (On Apple-Silicon macOS the OS monotonic clock may tick through sleep, making the report best-effort there — noted in the code; enforcement can never be worse than the wall clock it replaces.)

Verification

Live feed: both families fully green (4/4, 7/7), so the probe is inert on a healthy network.
MESHWEAVER_SKIP_NUGET=1 → explicit Skipped, not a silent pass.
dotnet build -c Release -warnaserror clean on Graph.Test, MemexTemplate.Test, Hosting.Monolith.Test (TestBase dependent).

No What's New entry: test-infrastructure only, no user-visible change.

🤖 Generated with Claude Code

… monotonic time Fixes #679 — two families hard-failed whenever nuget.org was unreachable, presenting as phantom CI flakes whose messages point at NuGet internals: - NuGetAssemblyResolverTest: each test now probes the api.nuget.org service index first and Assert.Skips with the cause when the feed is down or unhealthy; the MESHWEAVER_SKIP_NUGET=1 opt-out becomes an explicit skip instead of a silent vacuous pass. - MemexTemplateGeneratorTest: same probe before spawning the generator (dotnet run's implicit restore is the hidden network edge), plus an NU1900/service-index detector for a feed that dies between probe and restore — that shape skips naming the outage; real generator defects still fail. Also fixes the related hazard from the same sweep: the MonolithMeshTestBase watchdog measured wall-clock, so a host suspension (laptop sleep, stalled runner) presented as a 5000+s 'hang' against a 720s cap. Deadlines are now enforced on a monotonic Stopwatch, and a wall-vs-monotonic divergence >10s is reported as [WATCHDOG-SUSPEND] naming the suspension window. Verified: both families green against the live feed (4/4, 7/7), env-var path yields an explicit Skip; Release -warnaserror clean on all touched projects. Closes #679 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Copilot

Pull request overview

This PR addresses #679 by making NuGet-feed-dependent tests report environmental outages as explicit skips (instead of misleading failures), and by updating MonolithMeshTestBase’s watchdog to enforce deadlines using monotonic time to avoid false “hang” reports caused by host suspension.

Changes:

Added per-test nuget.org reachability probes to skip NuGet E2E tests with a clear reason, and made MESHWEAVER_SKIP_NUGET=1 produce an explicit skip instead of a silent pass.
Added the same reachability probe to the Memex template generator tests and added output-based detection to skip when the feed fails between the probe and dotnet run restore.
Updated the Monolith test watchdog to use Stopwatch (monotonic) for deadline enforcement and log a [WATCHDOG-SUSPEND] diagnostic when wall vs monotonic time diverges significantly.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File	Description
test/MeshWeaver.MemexTemplate.Test/MemexTemplateGeneratorTest.cs	Adds a nuget.org probe + outage-specific skip logic around the generator’s implicit restore path.
test/MeshWeaver.Graph.Test/NuGetAssemblyResolverTest.cs	Adds a per-test nuget.org probe and converts the `MESHWEAVER_SKIP_NUGET` behavior into an explicit skip.
src/MeshWeaver.Hosting.Monolith.TestBase/MonolithMeshTestBase.cs	Enforces watchdog deadlines using monotonic time and logs suspected host suspension windows.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+        catch (Exception ex) when (ex is HttpRequestException or TaskCanceledException)
+        {
+            Assert.Skip($"api.nuget.org unreachable ({ex.GetType().Name}: {ex.Message}) — the " +
+                "generator's implicit `dotnet run` restore needs the live feed; a network outage " +
+                "presents as a SKIP, not a phantom test failure (#679)");
+        }


+        catch (Exception ex) when (ex is HttpRequestException or TaskCanceledException)
+        {
+            Assert.Skip($"api.nuget.org unreachable ({ex.GetType().Name}: {ex.Message}) — this " +
+                "end-to-end test needs the live feed; a network outage presents as a SKIP, not a " +
+                "phantom test failure (#679)");
+        }


github-actions · 2026-08-07T06:36:57Z