benchmarking/locust: capture cluster hardware facts and density frontiers in runner by Nishanth29 · Pull Request #1603 · agent-substrate/substrate

Nishanth Kotla (Nishanth29) · 2026-09-10T23:07:18Z

What this PR does

In alignment with the Actor Density Benchmark Specs, this PR updates benchmarking/locust/runner.py to auto-discover cluster capacity, record hardware density frontiers (both client-side and server-side A/P ratios), and harvest server-side Prometheus metrics into stats.jsonl and server_summary.json.

Proposed Changes

Cluster Hardware Discovery (runner.py, locust.yaml):
- Queries the Kubernetes API for allocatable cores, RAM, node count, and worker pods (with env var overrides).
- Adds read-only RBAC (nodes, pods) for the Locust service account in locust.yaml.
Client & Server A/P Bin-Packing & Density Frontiers:
- Client-side (trial_summary): Computes actors_per_node, actors_per_vcpu, actors_per_gb_ram, and steady-state A/P demand percentiles from stats_history.csv.
- Server-side (server_summary): Gathers true physical worker assignment (ate_workerpool_workers), host Linux kernel PSI stalls (CPU/Mem/IO), and snapshot performance from in-cluster Prometheus.
- Appends entries to stats.jsonl and outputs server_summary.json while keeping status.json schema strictly unchanged.
Defensive Prometheus Harvester (server_telemetry.py):
- Uses only the Python standard library with timeouts and graceful fallback if Prometheus is unreachable.

How this was tested

Unit Tests: Added 8 unit tests in benchmarking/locust/unit_tests/test_server_telemetry.py (all passed).
Cluster Validation: Verified on a benchmark cluster with multi-user glutton runs:
- Confirmed stats.jsonl contains trial_summary (hardware facts + client frontiers) and server_summary.
- Confirmed server_summary.json is populated with Prometheus metrics.
- Confirmed status.json minimal schema is preserved.

References

Agent Substrate: Actor Density Benchmark Specs
Tests pass
Appropriate changes to documentation are included in the PR

…iers in runner

Max Smythe (maxsmythe)

Thanks for this! Left a few initial comments.

Max Smythe (maxsmythe) · 2026-09-11T10:05:38Z

    if with_boomer:
-        boomer_cmd = [BOOMER_BINARY, "--user-class", Path(test_file(args.file)).stem]
+        boomer_cmd = [BOOMER_BINARY]
+        if "boomer-glutton" not in os.path.basename(BOOMER_BINARY):


Is all this switching to avoid breaking changes? Shouldn't the binary be bundled in the container? May be able to just straight rename the binary.

yeah, my benchmark cluster was still running an older image from before the #1295 rename, so I needed
these to get my trials running against it. None of it applies to a normal run since the Dockerfile
bundles /app/boomer-worker. Reverting to that and dropping the env knobs.

Max Smythe (maxsmythe) · 2026-09-11T10:07:11Z

+    return int(float(mem_str))
+
+
+def get_cluster_hardware_facts(logs: TextIO | None = None) -> dict[str, any]:


I think this batch of code is large enough and focused enough to warrant its own file.

will move it into its own file

Max Smythe (maxsmythe) · 2026-09-11T10:09:29Z

+        "allocatable_ram_gb": None,
+        "worker_pod_count": None,
+    }
+    # 1. Environment variables override


Why do we need overrides?

I added them as a fallback in case the runner couldn't read the cluster, but this PR adds the RBAC for
that anyway so they aren't really needed. Removing them.

Max Smythe (maxsmythe) · 2026-09-11T10:10:43Z

+    ):
+        return facts
+
+    # 2. In-cluster HTTP API or local kubectl discovery


I think K8s has a Python client we can use for this?

oh yeah... I went with urllib because the locust image is distroless and I was trying to avoid
adding a dependency, but we already install pip packages in there anyway, so that isn't really a
concern. will be switching to the client.

Max Smythe (maxsmythe) · 2026-09-11T10:11:45Z

+                    except Exception:
+                        pass
+        elif shutil.which("kubectl"):
+            # Local workstation fallback via kubectl


Why do we need this? Also I think we may get standard auth for free w/a client

that was just so I could run runner.py from my workstation, but you're right that the client handles
that for us. Dropping the kubectl path.

Max Smythe (maxsmythe) · 2026-09-11T10:13:38Z

                else:
                    stats_generated = jsonl_path.exists()
+                    if stats_generated:
+                        facts = get_cluster_hardware_facts(logs)


for truly large clusters, we may want the ability to disable this (hitting every node may take a long time)

good call, will be adding a --cluster-facts / --no-cluster-facts flag (default on) so it can be skipped
entirely.

Nishanth Kotla (Nishanth29) · 2026-09-12T00:13:07Z

Thanks for the review, Max. All fixes are in the latest commit.

Reverted the boomer switching (that only existed because my cluster was on a
pre-#1295 image), moved discovery into cluster_facts.py, dropped the env
overrides and the kubectl fallback, and switched to the K8s client.

Added a --cluster-facts / --no-cluster-facts flag, on by default, so the
node and pod listing can be skipped entirely on large clusters.

Other things I did along the way:

The ClusterRole was only bound to the default SA, but automated runs use
benchmark-runner, so they'd have been denied and quietly recorded nulls.
Bound both, and added the role to the runner Job template so it doesn't
depend on the locust Deployment being applied.
worker_pod_count is now null when the worker pool isn't discoverable. It
used to fall back to the node count, which was indistinguishable from a real
reading in the output and skewed the A/P ratios derived from it.
Added machine_type to raw_configuration. Without it, runs on different
hardware look identical in the output.

Tests

benchmarking/locust/unit_tests/test_cluster_facts.py — 30 tests covering
node capacity parsing, machine type, worker pod counting, 403 handling, the
--cluster-facts flag, and the trial_summary schema.
benchmarking/locust/unit_tests/test_server_telemetry.py — 9 tests covering
percentiles, steady-state window detection, the Prometheus query guards, and
packing math.

Run with:

pip install -r benchmarking/locust/requirements.txt
python3 -m unittest discover -s benchmarking/locust/unit_tests

Also validated on a live cluster with glutton runs.

Nishanth Kotla (Nishanth29) added 2 commits September 10, 2026 16:50

benchmarking/locust: capture cluster hardware facts and density front…

8c11e38

…iers in runner

benchmarking/locust: add server-side telemetry harvesting

dc4ddaf

Max Smythe (maxsmythe) requested changes Sep 11, 2026

View reviewed changes

benchmarking/locust: address review feedback

aa26f04

		return int(float(mem_str))


		def get_cluster_hardware_facts(logs: TextIO \| None = None) -> dict[str, any]:

Uh oh!

Conversation

Nishanth Kotla (Nishanth29) commented Sep 10, 2026

What this PR does

Proposed Changes

How this was tested

References

Uh oh!

Max Smythe (maxsmythe) left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Nishanth Kotla (Nishanth29) commented Sep 12, 2026

Tests

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants