Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions Directory.Build.targets
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,20 @@
<NoWarn>$(NoWarn);ASHLAREXP002</NoWarn>
</PropertyGroup>

<!-- Directory.Build.props adds versionless System.Text.Encodings.Web and
System.Text.RegularExpressions references so central package management can lift the
vulnerable transitives. That file is imported before the project body, so it cannot see a
project that sets ManagePackageVersionsCentrally to false. Leaving the versionless items
in place is NU1015. Metadata conditions are illegal outside a target (MSB4190/MSB4191),
so the drop runs here, before NuGet collects references. A project that pins its own
Version keeps that pin. -->
<Target Name="AshlarDropVersionlessAuditOverrides" BeforeTargets="CollectPackageReferences">
<ItemGroup Condition="'$(ManagePackageVersionsCentrally)' == 'false'">
<PackageReference Remove="System.Text.Encodings.Web" Condition="'%(PackageReference.Version)' == ''" />
<PackageReference Remove="System.Text.RegularExpressions" Condition="'%(PackageReference.Version)' == ''" />
</ItemGroup>
</Target>

<!-- NetStandard compatibility shims (Unity 2019+): required members, init-only, etc.
Evaluated statically (not inside a <Target>) so IDE/language-server design-time builds
also see the polyfills; a Target-added <Compile> item is invisible to design-time builds,
Expand Down
9 changes: 8 additions & 1 deletion application/src/Ashlar.API/Endpoints/AshlarEndpoints.cs
Original file line number Diff line number Diff line change
Expand Up @@ -1481,7 +1481,8 @@ private static bool TryResolveProductTenant(
return summary;

var safeToForward = IsDevelopment()
|| string.Equals(last.IssueType, BarrierContextMissingIssueType, StringComparison.Ordinal);
|| string.Equals(last.IssueType, BarrierContextMissingIssueType, StringComparison.Ordinal)
|| string.Equals(last.IssueType, NoWorkPlaceholderIssueType, StringComparison.Ordinal);
return safeToForward
? $"{summary}: {last.Description}"
: $"{summary}: see escalations ({last.IssueType})";
Expand All @@ -1493,6 +1494,12 @@ private static bool TryResolveProductTenant(
/// </summary>
private const string BarrierContextMissingIssueType = "BARRIER_CONTEXT_MISSING";

/// <summary>
/// Mirrors the orchestrator's no-work escalation. The description is static (no exception
/// text), so it is safe to put on the receipt outside Development.
/// </summary>
private const string NoWorkPlaceholderIssueType = "NO_WORK";

// The escalation issue type is the closest thing to a canonical code the orchestrator
// emits (BarrierContextMissingException escalates under its own ErrorCode).
private static string? ResolveOrchestrationErrorCode(OrchestrationResult result) =>
Expand Down
2 changes: 1 addition & 1 deletion docs/TesterQuickstart.md
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,7 @@ $r | Select-Object taskId, success, summary

The route is `POST /api/copilot/task` (`application/src/Ashlar.API/Endpoints/AshlarEndpoints.cs`); the body is `CopilotTaskRequest(string Task, int AuditCount = 25)` (`application/src/Ashlar.API/Endpoints/CopilotTaskRequest.cs`). The response is `CopilotTaskResponse`: `taskId`, `tenantId` (`default` unless you send `X-Ashlar-Tenant`), `success`, `summary` (for example `1 agent(s) executed`), `output` (per-agent results), `isTrustPaused`, and `recentAudit`. Two things to know when reading it:

- With the mock provider the `output` text is deterministic scaffolding (a "fallback decomposition"), not model output. The point of this page is the **record**, not the prose. To route to a real model, set `ASHLAR_OLLAMA_BASE_URL` / `OPENAI_API_KEY` instead of `ASHLAR_ALLOW_MOCK` (see `docs/Configuration.md`).
- With the mock provider the `output` text is deterministic scaffolding (a "fallback decomposition"), not model output. The point of this page is the **record**, not the prose. When that scaffolding is only the no-work placeholder (`placeholder: true`, no specialized agent), `success` is false and the audit disposition is `Failure`. To route to a real model, set `ASHLAR_OLLAMA_BASE_URL` / `OPENAI_API_KEY` instead of `ASHLAR_ALLOW_MOCK` (see `docs/Configuration.md`).
- ~~`recentAudit` holds the entries recorded **before** this task (the handler snapshots the log, then records the task), so it is `[]` on your first call and shows the previous task on the second.~~ **[Fixed 2026-09-20]** `recentAudit` now includes this task's audit entry — the handler logs the task entry before reading the recent audit, so your first call shows the `CopilotTask` entry immediately.

Now read the trail the task left:
Expand Down
4 changes: 2 additions & 2 deletions docs/knowledge-graph.json
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
"operator_vars_missing_from_configuration_reference": 17,
"operator_vars_absent_from_all_documentation": 0,
"test_files_declaring_facts": 811,
"declared_facts": 5207,
"declared_facts": 5209,
"workflows": 61
},
"operator_vars_missing_from_configuration_reference": [
Expand Down Expand Up @@ -8088,7 +8088,7 @@
"file": "src/Ashlar.Tests.Orchestration/OrchestrationOrchestratorGapCoverageTests.cs",
"project": "Ashlar.Tests.Orchestration",
"namespace": "Ashlar.Tests.Orchestration",
"declared_facts": 19,
"declared_facts": 21,
"in_cert_gate_filter": false
},
{
Expand Down
2 changes: 1 addition & 1 deletion docs/knowledge-graph.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ remembered to update — staleness here is a red build.
| missing from `docs/Configuration.md` | 17 |
| **mentioned in no document at all** | **0** |
| test files declaring xUnit facts | 811 |
| declared facts | 5207 |
| declared facts | 5209 |
| workflows | 61 |

## Namespaces cert-gate selects
Expand Down
2 changes: 1 addition & 1 deletion docs/sdk.md
Original file line number Diff line number Diff line change
Expand Up @@ -106,7 +106,7 @@ sdk.RegisterBrick<OWASPScannerBrick>();
sdk.RegisterBrick<MySecurityBrick>();
```

Bricks are resolved via dependency injection when instantiated. Ensure required services (e.g. `IProviderFactory`) are registered.
Resolve a registered brick with `IBrickRegistry.GetBrick`. `AddAshlarBrick<T>()` adds the type to that registry; it does not register `T` as its own service. Ensure required services (e.g. `IProviderFactory`) are registered.

## RegisterAgent&lt;T&gt;

Expand Down
32 changes: 32 additions & 0 deletions src/Ashlar.Orchestration/Agents/GenericAgent.cs
Original file line number Diff line number Diff line change
Expand Up @@ -68,3 +68,35 @@ protected override Task OnShutdownAsync(CancellationToken cancellationToken)
}
}

/// <summary>
/// A no-work fallback is still a successful agent call (the transport delivered it). The
/// orchestration result is a different flag: when every output is one of these, the run
/// performed no work and must not report success.
/// </summary>
internal static class NoWorkPlaceholder
{
internal static bool Is(object? output)
{
if (output is null)
return false;

var property = output.GetType().GetProperty("Placeholder");
return property is { CanRead: true, PropertyType: var type }
&& type == typeof(bool)
&& property.GetValue(output) is true;
}

internal static bool Only(IEnumerable<object> outputs)
{
var any = false;
foreach (var output in outputs)
{
any = true;
if (!Is(output))
return false;
}

return any;
}
}

19 changes: 17 additions & 2 deletions src/Ashlar.Orchestration/Coordination/Orchestrator.cs
Original file line number Diff line number Diff line change
Expand Up @@ -490,12 +490,27 @@ await _barrierGuard.AssertValidForInvocationAsync(
: null;
var integratedOutput = _outputIntegrator.Integrate(decomposition.Agents, outputs);

// GenericAgent's transport result stays Success (the call completed). A run whose every
// output is that no-work placeholder did not do the task; reporting it as success is
// the silent green receipt. Escalate under a static code so the API summary can say
// why outside Development, the same way a missing barrier does.
var noWork = NoWorkPlaceholder.Only(outputs.Values);
if (noWork)
{
_escalationManager.EscalateIssue(
"NO_WORK",
"No specialized agent performed work; the result is a placeholder.",
EscalationSeverity.High);
}

var success = integratedOutput.IsValid && !noWork;

// Step 6: Shutdown all agents
await _lifecycleManager.ShutdownAllAsync(cancellationToken);

if (rootSpanId != null)
{
_metrics?.EndSpan(rootSpanId, integratedOutput.IsValid, new Dictionary<string, string>
_metrics?.EndSpan(rootSpanId, success, new Dictionary<string, string>
{
["agentCount"] = decomposition.Agents.Count.ToString(),
["conflictCount"] = conflicts.Count.ToString(),
Expand All @@ -505,7 +520,7 @@ await _barrierGuard.AssertValidForInvocationAsync(

return new OrchestrationResult
{
Success = integratedOutput.IsValid,
Success = success,
IntegratedOutput = integratedOutput,
Decomposition = decomposition,
Conflicts = conflicts,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -558,6 +558,60 @@ public async Task OrchestrateAsync_surfaces_missing_barrier_context_as_a_distinc
Times.Never);
}

[Fact]
public async Task OrchestrateAsync_reports_failure_when_every_output_is_a_no_work_placeholder()
{
var transportMock = new Mock<IAgentTransport>();
transportMock
.Setup(t => t.SendAsync(It.IsAny<AgentInvocationRequest>(), It.IsAny<CancellationToken>()))
.ReturnsAsync(new AgentResult(
Success: true,
Output: new { Placeholder = true, Output = "No specialized agent performed work" }));

var decomposition = new DecompositionResult
{
OriginalRequest = "what is 2 + 2",
Agents = [new AgentSpawnSpec { AgentId = "fallback-1", Domain = "Gameplay", Goal = "Handle request" }],
};

var sut = CreateOrchestrator(CreateArchitectMock(decomposition).Object, transportMock.Object);
var result = await sut.OrchestrateAsync("what is 2 + 2");

result.Success.Should().BeFalse();
result.IntegratedOutput!.AgentOutputs.Should().ContainKey("fallback-1");
var escalation = result.Escalations.Should().ContainSingle().Subject;
escalation.IssueType.Should().Be("NO_WORK");
escalation.Description.Should().Be("No specialized agent performed work; the result is a placeholder.");
}

[Fact]
public async Task OrchestrateAsync_stays_successful_when_a_real_output_sits_beside_a_placeholder()
{
var transportMock = new Mock<IAgentTransport>();
transportMock
.Setup(t => t.SendAsync(It.Is<AgentInvocationRequest>(r => r.AgentName == "fallback-1"), It.IsAny<CancellationToken>()))
.ReturnsAsync(new AgentResult(Success: true, Output: new { Placeholder = true, Output = "no work" }));
transportMock
.Setup(t => t.SendAsync(It.Is<AgentInvocationRequest>(r => r.AgentName == "real-1"), It.IsAny<CancellationToken>()))
.ReturnsAsync(new AgentResult(Success: true, Output: new { ok = true }));

var decomposition = new DecompositionResult
{
OriginalRequest = "mixed",
Agents =
[
new AgentSpawnSpec { AgentId = "fallback-1", Domain = "Gameplay", Goal = "Handle request" },
new AgentSpawnSpec { AgentId = "real-1", Domain = "Security", Goal = "Check" },
],
};

var sut = CreateOrchestrator(CreateArchitectMock(decomposition).Object, transportMock.Object);
var result = await sut.OrchestrateAsync("mixed");

result.Success.Should().BeTrue();
result.Escalations.Should().BeEmpty();
}

[Fact]
public async Task OrchestrateAsync_defaults_to_floor_barrier_when_context_missing_and_not_required()
{
Expand Down
84 changes: 84 additions & 0 deletions tests/scripts/central-management-opt-out.test.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
#!/usr/bin/env bash
# Directory.Build.props injects versionless PackageReferences for the two audit overrides
# (System.Text.Encodings.Web, System.Text.RegularExpressions). A project that sets
# ManagePackageVersionsCentrally to false has no central version for them, and restore
# fails NU1015. The props file is imported before the project body, so the skip lives in
# Directory.Build.targets, which runs after that property is set: drop only the versionless
# copies of those two packages. A project that leaves central management on still receives
# them. A project that pins its own Version keeps that pin.
#
# Run: bash tests/scripts/central-management-opt-out.test.sh
# Pure bash + python3: no network, no dotnet, no container.

set -uo pipefail

ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
PASS=0
FAIL=0
ok() { PASS=$((PASS + 1)); echo " ok — $1"; }
bad() { FAIL=$((FAIL + 1)); echo " FAIL — $1"; echo " $2"; }

python3 - "${ROOT}/Directory.Build.props" "${ROOT}/Directory.Build.targets" << 'PY'
import sys
import xml.etree.ElementTree as ET

NS = {"m": "http://schemas.microsoft.com/developer/msbuild/2003"}
AUDIT = ("System.Text.Encodings.Web", "System.Text.RegularExpressions")

def load(path):
# The repo files omit the MSBuild namespace. Keep the parser strict about that.
text = open(path, encoding="utf-8").read()
if "schemas.microsoft.com/developer/msbuild/2003" in text:
raise SystemExit(f"unexpected default namespace in {path}")
return ET.parse(path).getroot()

props = load(sys.argv[1])
targets = load(sys.argv[2])

injected = []
for group in props.findall("ItemGroup"):
for ref in group.findall("PackageReference"):
name = ref.get("Include")
if name in AUDIT:
injected.append((name, ref.get("Version"), group.get("Condition")))

if [name for name, _, _ in injected] != list(AUDIT):
raise SystemExit(f"props no longer injects exactly the two audit overrides: {injected}")
if any(version for _, version, _ in injected):
raise SystemExit(f"audit overrides must stay versionless in props: {injected}")
print("ok props injects the two versionless audit overrides")

drops = [
t for t in targets.findall("Target")
if t.get("Name") == "AshlarDropVersionlessAuditOverrides"
]
if len(drops) != 1:
raise SystemExit(f"expected one AshlarDropVersionlessAuditOverrides target, found {len(drops)}")
target = drops[0]
before = target.get("BeforeTargets") or ""
if "CollectPackageReferences" not in before:
raise SystemExit(f"drop target must run before NuGet collects references: {before}")
removes = []
for group in target.findall("ItemGroup"):
cond = group.get("Condition") or ""
for ref in group.findall("PackageReference"):
removes.append((cond, ref.get("Remove"), ref.get("Condition") or ""))
if not removes or "'$(ManagePackageVersionsCentrally)' == 'false'" not in removes[0][0]:
raise SystemExit(f"Remove group is not gated on central management off: {removes}")
by_name = {name: cond for _, name, cond in removes}
if set(by_name) != set(AUDIT):
raise SystemExit(f"Remove must name exactly the two audit overrides: {removes}")
for name, item_cond in by_name.items():
if item_cond != "'%(PackageReference.Version)' == ''":
raise SystemExit(f"{name} remove must keep a project that pins Version: {item_cond}")
print("ok targets drops only the versionless audit overrides when central management is off")
PY
PY_RC=$?
if [[ "${PY_RC}" -eq 0 ]]; then
ok "versionless audit overrides are skipped when central management is off"
else
bad "central-management opt-out" "checker exited ${PY_RC}"
fi

echo "central-management-opt-out: ${PASS} passed, ${FAIL} failed"
[[ "${FAIL}" -eq 0 ]]
3 changes: 3 additions & 0 deletions tests/uat/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -143,3 +143,6 @@ meant remapping dependency edges, execution order, outputs and the integrator co

Fixing any one of the three alone looks like success to a check that counts HTTP 200s. Tier 10 asks
whether the **work reached the record**, which is why it kept failing until all three were gone.
The mock fallback still performs no work, so a finished run is `success: false` with the no-work
summary. Tier 10 counts those receipts. It still fails a body that says `cannot execute from state
Executing`, which is the shared-agent crash and not the placeholder.
13 changes: 10 additions & 3 deletions tests/uat/tier0-2.sh
Original file line number Diff line number Diff line change
Expand Up @@ -226,10 +226,17 @@ R=$(curl -s --max-time 300 "$API/api/copilot/task" -H "Content-Type: application
TASK_S=$(( $(now) - t0 ))
echo "$R" >"$OUT/task.json"
TASK_ID=$(printf '%s' "$R" | grep -o '"taskId":"[^"]*"' | head -1 | cut -d'"' -f4)
if [ -n "$TASK_ID" ] && printf '%s' "$R" | grep -q '"success":true'; then
result 1 task-submit PASS "${TASK_S}s, taskId=$TASK_ID, success=true"
# The mock provider's reply is not decomposition JSON, so the run is the Gameplay
# fallback and GenericAgent performs no work. success true here was the silent
# green receipt. The receipt is a task id, success false, the static no-work
# summary, and a Failure audit disposition.
if [ -n "$TASK_ID" ] \
&& printf '%s' "$R" | grep -q '"success":false' \
&& printf '%s' "$R" | grep -q 'No specialized agent performed work' \
&& printf '%s' "$R" | grep -q '"disposition":"Failure"'; then
result 1 task-submit PASS "${TASK_S}s, taskId=$TASK_ID, success=false (no-work placeholder)"
else
result 1 task-submit FAIL "${TASK_S}s, no taskId/success in: $(printf '%s' "$R" | head -c 300)"
result 1 task-submit FAIL "${TASK_S}s, expected taskId plus success false and the no-work summary, got: $(printf '%s' "$R" | head -c 300)"
fi

say "1.4 first-task audit is immediately visible (empty-first-audit bug fixed 2026-09-20)"
Expand Down
Loading
Loading