The auto-execute tests asked for an agent that has not existed since May by rbuergi · Pull Request #1080 · Systemorph/MeshWeaver

rbuergi · 2026-08-10T07:20:05Z

The symptom

OrleansAutoExecuteTest.AutoExecute_CreatesResponseCell_And_CompletesExecution passed alone but failed in bulk with:

Selected agent 'Orchestrator' was not found among the available agents ([Agent/Assistant … Agent/ThreadNamer])

Alone-passes + in-suite-fails reads like a shared-cluster ordering race. It isn't one.

What was actually wrong

Orchestrator was renamed to Assistant in c31fd04 (2026-05-23). No such agent has shipped since. OrleansThreadStreamingTest already says so outright in a comment. Every round these tests drove could therefore only produce the terminal "agent not found" error — alone or in bulk.

The order dependence came entirely from assertions that never looked at the round's outcome:

Match(m => !string.IsNullOrEmpty(m.Text)) matched the first emission carrying any text — the "Allocating agent..." placeholder the framework writes when it allocates the response cell. Whether the subscription caught that emission or a later one is pure scheduling luck. Running alone, the observed response text was literally Allocating agent...: the test passed without any agent having run.
Match(t => t is { IsExecuting: false } && t.Messages.Count >= 2) is satisfiable in the pre-execution window as well as after the round, so "wait for execution to settle" could return before execution started.
The sibling test screened for placeholder prefixes, which the "agent not found" error does not start with — so it asserted an error message as "final text".

Product vs. test — the convergence hypothesis is falsified

The suspicion was that AgentChatClient treats a partially-converged catalog as authoritative. It does not apply here. With a correct terminal-condition assertion the test fails deterministically in 1 second running alone, and the error names the complete 13-agent built-in catalog — the identical list seen in the bulk failure. The catalog is fully converged in both cases; Orchestrator is simply not in it.

AgentChatClient is behaving correctly and deliberately: an explicit-but-unresolvable selection surfaces a named terminal error rather than silently running a different agent (the documented fix for the prod wrong-agent confusion). Nothing was relaxed there.

The fix

Point both tests at Agent/Assistant — the shipped default (isDefault: true), in the full node PATH form the picker stores — and wait on the framework's own round-done gate: the response cell's terminal Status write, then assert the echo agent's actual output. The IsExecuting == false check moves after the terminal write, where it is a real observed condition instead of a coin flip.

AutoExecuteFlowTest (MeshWeaver.Threading.Test) carried the identical stale name and the same "any non-empty text" assertion; fixed the same way.

Deterministic pre-fix failure (verified by reverting)

With the terminal-Status assertion in place and agentName: "Orchestrator" restored, running alone:

Failed AutoExecute_CreatesResponseCell_And_CompletesExecution [1 s]
  Expected "Selected agent 'Orchestrator' was not found among the available agents
  ([Agent/Assistant, Agent/Researcher, Agent/TrainingSim, Agent/Tutor, Agent/Worker,
  Agent/ExecutiveAssistant, Agent/LogTriage, Agent/NotificationTriage, Agent/EmailRouter,
  Agent/PullRequestWriter, Agent/DescriptionWriter, Agent/NodeInitializer, Agent/ThreadNamer]).
  It may have been moved, renamed, or is not available in this context — pick another agent
  from the list." to contain "Echo:" because the echo agent should have written the response text.

Post-fix both tests report Echo: 2 messages received. — the agent actually ran.

Verification

MeshWeaver.Hosting.Orleans.Test full project, DOTNET_PROCESSOR_COUNT=4: 148/148 passed
MeshWeaver.Threading.Test full project: 128/128 passed
dotnet build -c Release -warnaserror clean for both projects

No What's New entry

This change touches only test files — no user-visible behaviour changes, so there is nothing a user can notice (AGENTS.md scopes Category: Fix to user-noticeable fixes). Same shape as #1051, a test-infrastructure-only PR that shipped without an entry.

🤖 Generated with Claude Code

`OrleansAutoExecuteTest` passed alone and failed in bulk with "Selected agent 'Orchestrator' was not found among the available agents". The catalog was never the problem: `Orchestrator` was renamed to `Assistant` in c31fd04, so no such agent has shipped since 2026-05-23. Every round these tests drove could only ever produce the terminal "agent not found" error. What made that LOOK like a race was the assertions, not the product: - `Match(m => !string.IsNullOrEmpty(m.Text))` matched the FIRST emission carrying any text — the "Allocating agent..." PLACEHOLDER the framework writes when it allocates the response cell. Whether the subscription caught the placeholder or a later emission was pure scheduling luck, so the same test passed alone (observed text: "Allocating agent...") and failed under load. It never once asserted that an agent ran. - `Match(t => t is { IsExecuting: false } && t.Messages.Count >= 2)` is satisfiable in the PRE-execution window as well as after the round, so the "wait for execution to settle" step could return before execution started. - The sibling test screened for placeholder PREFIXES, which the "agent not found" error does not start with — so it asserted an error as "final text". Point both tests at `Agent/Assistant` (the shipped default, in the full node PATH form the picker stores) and wait on the framework's own round-done gate: the response cell's terminal `Status` write, then the echo agent's actual output. `AutoExecuteFlowTest` carried the identical stale name and the same "any non-empty text" assertion; fixed the same way. Verified: with the terminal-Status assertion in place and `Orchestrator` restored, the test fails DETERMINISTICALLY in 1 s running alone, against a fully converged 13-agent catalog — which is what rules out a convergence race. Post-fix both tests report `Echo: 2 messages received.` Orleans 148/148, Threading 128/128, Release -warnaserror clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Copilot

Pull request overview

This PR fixes several auto-execute/threading tests that were passing despite never successfully executing an agent round, due to a stale agent name (Orchestrator → Agent/Assistant) and assertions that could match pre-execution placeholder output.

Changes:

Update tests to select the shipped built-in agent by node path (Agent/Assistant) instead of the removed Orchestrator name.
Replace non-deterministic “any non-empty text / IsExecuting == false” waits with predicates that gate on round completion and validate real agent output (Echo:).

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File	Description
test/MeshWeaver.Threading.Test/AutoExecuteFlowTest.cs	Updates portal flow test to use `Agent/Assistant` and assert on terminal completion and echo output.
test/MeshWeaver.Hosting.Orleans.Test/OrleansAutoExecuteTest.cs	Fixes Orleans auto-execute tests by using `Agent/Assistant` and waiting on terminal message status before asserting output.

Suppressed comments (1)

test/MeshWeaver.Hosting.Orleans.Test/OrleansAutoExecuteTest.cs:184

Same as above: matching only Completed turns fast terminal failures into timeouts. Waiting for any terminal status and then asserting Completed makes failures deterministic and keeps the error message visible in the assertion output.

        var msg = await GetHubContent<ThreadMessage>(client, responsePath)
            .Should().Within(30.Seconds())
            .Match(m => m is { Status: ThreadMessageStatus.Completed });
        msg!.Text.Should().Contain("Echo:",
            "the streamed agent output must reach the RESPONSE grain, not the thread grain");

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

        var response = await ThreadFlow.ReadMessage(client, threadPath, responseMsgId,
-            m => !string.IsNullOrEmpty(m.Text) && m.Status != ThreadMessageStatus.Streaming).Should().Within(30.Seconds()).Emit();
-        response.Text.Should().NotBeNullOrEmpty("agent should have written response");
+            m => m.Status == ThreadMessageStatus.Completed).Should().Within(30.Seconds()).Emit();
+        response.Text.Should().Contain("Echo:", "the echo agent should have written the response");


        var response = await GetHubContent<ThreadMessage>(client, responsePath)
            .Should().Within(30.Seconds())
-            .Match(m => !string.IsNullOrEmpty(m?.Text));
-        response!.Text.Should().NotBeNullOrEmpty("agent should have written response text");
+            .Match(m => m is { Status: ThreadMessageStatus.Completed });
+        response!.Text.Should().Contain("Echo:", "the echo agent should have written the response text");


github-actions · 2026-08-10T07:50:48Z