A starved security-policy read leaves CreateNodeRequest hanging for the full timeout instead of failing fast #1446

Found while investigating a CI failure in SourceDiscoveryUnavailableTest (fixed test-side in #1427). The test artifact is fixed; the product behaviour underneath is not, and it is worth a look on its own.

What happens

Creating a node runs the security layer's policy and grant reads, which are namespace:-shaped queries over the target subtree:

namespace:{…}/Source          id:_Policy   nodeType:PartitionAccessPolicy
namespace:{…}/Source/_Access               nodeType:AccessAssignment

When those reads starve — the cross-silo case this suite exists for (#1218) — the create does not fail fast. It sits waiting. Observed in the hub snapshot at teardown:

Hub portal/nodeops-… RunLevel=Started Disposal=Pending
  Queue(buffer=2, …) Executing(CreateNodeRequest, 33014ms)

33 seconds, still executing, on reads that could never answer.

Why it matters beyond the test

The faulted-query eviction already works — the log shows both queries evicted so the next GetQuery opens a fresh upstream:

its Replay(1) had latched the terminal error and would have replayed it to every future subscriber for the life of the process

So the query recovers. The in-flight caller does not — it is still blocked on the read that faulted. In a real cross-silo starvation, node creation into the affected namespace would hang this way for every caller until something above it times out.

Suggested shape

A starved security read is an availability failure, and the codebase already has the vocabulary for exactly this distinction — SourceDiscoveryUnavailableException / CompilationStatus.Unavailable were introduced (#1218) so a starved source read is not misreported as a compile verdict. The same reasoning applies here: a create whose permission check could not be established should fail fast and say so, rather than block.

Repro

Not currently reproducible from the suite — #1427 narrowed the stand-in to source discovery precisely so the security reads stop starving. To see it again, widen StarveMarker in SourceDiscoveryUnavailableTest to match the namespace alone (drop the nodeType:Code clause) and watch the CreateNode at the top of the test.

🤖 Generated with Claude Code