test: cover the request contract of every guest-agent, KMS and gateway RPC

[kvinwang]

Stacked on #841.

Problem

The per-method RPC cases in #841 send one valid request and one wrong-typed value for the first field. Out-of-range integers, oversized strings, bad protobuf framing and truncated bodies were never sent to any guest-agent, KMS or gateway RPC.

Change

One harness (api_contract_matrix.py + api-contract-matrix-case.py) drives every method of a service from catalog/api-inventory.json with per-field, per-representation and per-method malformed input. It asserts six invariants: no 5xx or dropped connection (L1), a structured error in the request's representation (L2, L6), the listener still answers afterwards (L3), rejections that neither echo nor scale with input (L4), and no call past its deadline (L5). api-contract-matrix-policy.json limits each method to full, rejection-only or skip, and a method without a policy fails its case.

Nine cases: 01-guest-os/15-request-contract (Tappd, DstackGuest, Worker, GuestApi), 03-kms/13-request-contract (KMS, Onboard), 04-gateway/11-request-contract (Gateway, Debug, Admin).

It found two product defects, fixed in #1232 and #1234:

{"invariant": "L1", "label": "DstackGuest.GetTlsKey|not_after=u64-max|json", "detail": "transport failure: curl: (52) Empty reply from server"}
{"invariant": "L4", "label": "DstackGuest.GetTlsKey|alt_names=long-1m|json", "detail": "rejection body is 1048664 bytes"}

The first is a process abort reachable from any application container through dstack.sock.

Verification

• tdxlab run cov841v-all-5860e60a4 (with fix(guest-agent): reject a certificate validity bound it cannot represent #1232 and fix(ra-rpc): bound the error text a failed call returns, and return it in the caller's representation #1234): all nine cases PASS.
• dstack-test verify-registry --plan test-suites: 0 findings, 369 scripted / 369 promoted.
• pytest test-suites/runner/tests: 40 passed. ruff check, ruff format --check, py_compile, reuse lint: clean.

[kvinwang]

tdxlab: 18/18 PASS

All the runnable new cases now execute on the tdxlab physical host against lease-owned fixtures. Run cov841v-all-5860e60a4, 1 worker, product binaries built from the revision under test with #1232 and #1234 merged in:

[1/18]  tc-gos-contract-001: PASS          [10/18] tc-kms-keyhier-001: PASS
[2/18]  tc-gos-contract-002: PASS          [11/18] tc-kms-keyhier-002: PASS
[3/18]  tc-gos-contract-003: PASS          [12/18] tc-kms-keyhier-003: PASS
[4/18]  tc-gos-contract-004: PASS          [13/18] tc-kms-keyhier-004: PASS
[5/18]  tc-gos-concurrency-001: PASS       [14/18] tc-kms-keyhier-005: PASS
[6/18]  tc-gos-concurrency-003: PASS       [15/18] tc-kms-keyhier-006: PASS
[7/18]  tc-gos-observability-slow-consumer: PASS   [16/18] tc-gw-contract-001: PASS
[8/18]  tc-kms-contract-001: PASS          [17/18] tc-gw-contract-002: PASS
[9/18]  tc-kms-contract-002: PASS          [18/18] tc-gw-contract-003: PASS
SWEEP_DONE rc=0

The six tc-kms-keyhier-* cases had only been validated against a mock KMS; this is their first run against a live one, and all eighteen promotions are re-recorded on this source_run with the offline-mock caveat removed.

tc-gos-concurrency-002 is deliberately not in the set: it refuses to PASS where a quote costs less than 50 ms, so it needs a physical TDX guest rather than the simulator fixture.

Three harness bugs the lab found and fixed along the way, none of them product issues — the echo invariant was an absolute ceiling where it should have been a scaling comparison, /proc/<pid>/fd is owner-only and aborted a case the fixture otherwise passed, and the gateway and onboard fixtures publish their listeners with the /prpc mount already in the URL, so prefixing it again addressed routes rocket does not serve. That last one is worth noting as evidence the assertions are real: every one of the 626 gateway admin vectors came back as rocket's default HTML 404 page, and the matrix reported exactly that — an unstructured rejection under the wrong content type, and a listener that did not answer a valid request afterwards.

[kvinwang]

Regression: 65/65 PASS on the chapters that share fixtures with the new cases

The new cases are additive, but they lease the same no-tee-dev and kms-ready fixtures as the existing guest-OS and KMS chapters, so I swept every scripted non-hardware case on those two profiles on tdxlab (65 cases, excluding the 19 new ones):

by_status {'PASS': 61, 'ERROR': 4}
failed ['tc-kms-auth-005', 'tc-kms-auth-006', 'tc-kms-auth-009', 'tc-kms-runtime-003']

The four errors are environmental, not a regression:

RuntimeError: Foundry contract submodules are not initialized:
  lib/forge-std/src/Test.sol,
  lib/openzeppelin-contracts-upgradeable/contracts/proxy/utils/UUPSUpgradeable.sol, ...

After git submodule update --init --recursive in dstack/kms/auth-eth and bun install --frozen-lockfile in the three Bun packages, all four pass:

[1/4] tc-kms-auth-005: PASS
[2/4] tc-kms-auth-006: PASS
[3/4] tc-kms-auth-009: PASS
[4/4] tc-kms-runtime-003: PASS
SWEEP_DONE rc=0

65/65.

Worth noting for #841 independently of this PR: prepare-hardware-run.sh provisions the Foundry submodules and the Bun packages, and prepare-run.sh does not — so the contract and auth-backend cases error on a plain preparation rather than reporting a named prerequisite. Either prepare-run.sh should provision them too, or those cases should report BLOCKED with the reason instead of a RuntimeError.

[kvinwang]

The hardware case found a real product defect, and then named it

tc-gos-concurrency-002 is the one case here that needs a physical TDX guest. Its first run on tdxlab failed with

15/43 concurrent DstackGuest.GetQuote under load calls failed:
  RemoteDisconnected: Remote end closed connection without response

and the case left it there. Every reading of that sentence is about the transport, and the fixture publishes a QEMU slirp forwarded host port into the guest, so the obvious suspect was the forwarder rather than the agent.

It was the agent. Driving the same 24-way load four ways against one lease-owned guest — over the forwarded port, over /var/run/dstack.sock from inside the guest, over the agent's own TCP listener, and through the fixture's socket bridge — drops 15 to 21 of 43 every time, and the guest's journal says why:

dstack-guest-agent.service: Watchdog timeout (limit 30s)!
dstack-guest-agent.service: Killing process 533 with signal SIGABRT

The systemd heartbeat is an HTTP Worker.Version request the agent sends to its own external listener, so a runtime parked on the global quote mutex cannot answer it. Control: 24-way Version under the same conditions answers 579 405 calls at a 2 ms p95 — so it is the blocking quote, not the concurrency.

Product fix: #1256. With it, all 43 are answered, Version holds at 2 ms p95, and four consecutive loads leave the agent in the process it started as.

Two corrections to the case itself, now committed here:

• It reads the agent's PID and start time from inside the guest over the fixture's ssh_argv, before and after the load, and checks that first — a dropped connection reads as a transport fault until you know the agent it was talking to was killed. Best-effort, so a fixture that publishes no guest access records nothing rather than blocking a case whose assertions are all measured over RPC.
• The worker count it reasons from was wrong. #[rocket::main] builds the runtime before load_config_figment runs, so the workers = 8 in dstack.toml never reaches it: the 2-vCPU guest was measured with two rocket-worker threads. The 24-thread load saturates either way, but the derivation should say the true one.

[kvinwang]

All 19 cases now pass on tdxlab, including the hardware-only one

tc-gos-concurrency-002 was the last case without a green run — it refuses to PASS where a quote costs less than 50 ms, so it needs a physical TDX guest, and against an image built from next it correctly fails by naming the agent restart it found.

Against a guest image built by mkosi from this branch plus #1256:

[1/1] tc-gos-concurrency-002: PASS
HWDONE rc=0

A trivial method stayed within its latency bound while the attestation path was saturated, in the agent process the case started with.

So the full set is 19/19 on tdxlab: 18 on lease-owned simulator and isolated-component fixtures (run cov841v-all-5860e60a4), and this one on a physical TDX guest (run cov841v-hw-ca26fb35e).