benchmarking: add an interference scenario (probe suspend/resume latency next to co-located load) #2806

Proposal

Add a benchmark scenario that measures how co-located actors slow suspend and resume. A few light probe actors cycle through suspend and resume while noise actors on the same node either write to their durable dirs or run the SWE-bench replay. Comparing each noise run with a probes-only run gives the slowdown, independent of the noise workload's own snapshot sizes.

Why

Nothing upstream measures this today, and it has caught real problems on micro-VM nodes (c3-highmem-192-metal, one node):

• With guest writes and snapshot staging on the same boot disk, probe suspend p95 went from 5.6 s with no noise to 59 s with 32 SWE-bench actors and to 120 s with 64 (v0.4). atelet, nodepath: separate per-actor snapshot directories under /var/lib/ate/snapshots #2746 lets snapshot staging have its own disk.
• On v0.5, the objectstore-plugin's 512Mi default upload budget (Limit memory used by concurrent snapshot uploads #2708) admits one GCS upload at a time per node, which put suspend p95 at 70 s / 134 s with 32 / 64 SWE-bench actors even with a dedicated snapshot disk (details in Bound compression and upload memory across concurrent snapshots #2553).

Shape

• A diskcontention boomer user class. The first --dc-probe-users VUs run Glutton probe actors, reported as Probe* rows. The rest run the noise chosen by --dc-noise-mode: disk (Glutton actors writing to their durable dir, optionally rate-capped, with or without fsync) or sweperf (the SWE-bench replay).
• The arms as tests.yaml entries run by the orchestrator (probes only, disk noise, SWE-bench noise), so they get the standard deploy and teardown, outputs, and server telemetry.
• Two small Glutton changes it needs: a per-request fsync option on WriteDisk (today fsync is a server-wide flag), and not counting calls canceled at shutdown as failures (the user class cancels its noise VUs on stop).
• Out of scope: the separate run, collect, and report scripts used for the investigation.

A reference implementation, including that separate harness, is in #2747 (now a draft). Without the harness it is about 1,600 lines, tests included.

Max Smythe (@maxsmythe), would you take this upstream? Happy to adjust the shape to fit the orchestrator.