benchmarking: add observability benchmarking for the OTel collector by JeffLuoo · Pull Request #749 · agent-substrate/substrate

Jeff Luo (JeffLuoo) · 2026-08-04T22:19:53Z

Part of #563

Substrate had no way to measure how much telemetry it sends. This PR adds one, and builds it from the parts the benchmark suite already has.

The managed collector sends its self-metrics to Cloud Monitoring only, and gives no data for each source. Thus you cannot see which component makes the volume. A second collector, the telemetry meter, counts the spans and the datapoints by service.name.

benchmarking/telemetry/meter.yaml — the meter. A tee: it counts the telemetry by service, then sends it on to the managed collector. Thus one run gives the volume for each service and the cost of the load together.
benchmarking/telemetry/README.md — the method: install, measure, read, remove.
hack/install-ate.sh --otlp-endpoint — sends all control plane telemetry to the meter. Each component reads the endpoint from the ate-otel-config ConfigMap through envFrom, and ate-controller copies the value to the ateom worker pods, thus one patch is sufficient for all of them.
benchmarking/workloads/deploy.sh --otlp-endpoint — the same for actor containers, which substrate configures separately.
benchmarking/monitoring.yaml — scrapes the meter and cAdvisor.
benchmarking/observability.md — prerequisites and the scenario ladder.
docs/dev/best-practices/otel-collector.md — the volume model and the size guidance, corrected for the ParentBased defaults from Define a production-ready default tracing policy #711.

Correction

The glutton ActorTemplate had no env. Substrate puts no OTLP configuration in an actor container, thus the exporter used the SDK default localhost:4317, no process listens on that address in the sandbox, and substrate discarded each actor span with no error message. The template now sets OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_RESOURCE_ATTRIBUTES.

ActorTemplate.spec is immutable, thus deploy.sh --deploy deletes the templates before it applies them.

How I verified this

On a GKE cluster with managed OTel. The sequence:

# 1. Start from a clean state.
./hack/install-ate.sh --delete-benchmarks
./hack/install-ate.sh --delete-ate-system

# 2. Install substrate from this branch.
./hack/install-ate.sh --deploy-ate-system

# 3. Install the meter and Prometheus.
kubectl apply -f benchmarking/telemetry/meter.yaml
kubectl apply -f benchmarking/monitoring.yaml

# 4. Send the control plane to the meter.
METER=http://telemetry-meter.benchmarking.svc.cluster.local:4317
./hack/install-ate.sh --deploy-ate-system --otlp-endpoint "${METER}"

# 5. Install the workloads and locust, with the actors on the meter also.
./benchmarking/deploy_locust.sh --deploy --worker-count 10 \
    --otlp-endpoint "${METER}"

# 6. Read the volume.
kubectl port-forward -n benchmarking svc/prometheus 9090:9090

Result

The idle floor, from one measurement on 10 workers:

Service	datapoints/min
`ate-controller`	234.8
`ateapi`	54.3
`atelet`	44.9
`atenet-router`	10.4
`ateom-gvisor`	5.2
`glutton-actor`	4.2
Total	353.7

The glutton-actor row is the correction above. Before it, actor telemetry could not arrive at all.

Positive control for the same window: otelcol_receiver_accepted_metric_points 10.5/s, and each otelcol_receiver_refused_* counter at zero. Thus the zero values in the table are true zeros, and not data that did not arrive.

Two values differ from the earlier manual measurement. ate-controller is now the largest source, and ateom-gvisor is no longer zero. The new imagecache counters from #831 are a probable cause. This difference is the reason to keep results out of the repository.

Traces

Trace volume follows the root sample decision, not the rate of each component. atelet and ateom-gvisor sent zero spans in each run, because they inherit the decision that ateapi makes and start no trace of their own. At idle each component is near zero: a trace needs an operation, but a metric pushes on a timer.

This run gives no span volume for a usual load. The actors crashed on suspend, and the failed retries made an artificial rate of 13.7k spans/s. That rate is not a measurement of substrate. It does show that the meter accepted the spans with no refusals.

Follow-up

A separate PR adds the ladder to automation/tests.yaml and writes the results as artifacts of each run. The open items are at the end of observability.md.

Tests pass
Appropriate changes to documentation are included in the PR

Krisztian F (krisztianfekete)

Added some comments! Do we want to make this repeatable locally (and in CI?) via a script?

Jeff Luo (JeffLuoo) · 2026-08-05T20:28:15Z

Added some comments! Do we want to make this repeatable locally (and in CI?) via a script?

Might do it in a follow-up and figure out the best way to trigger it. Running it against each CL might be expensive, can consider running it as a daily job..? Can connect with people who work on benchmarking and CI/CD to discuss about it.

Jeff Luo (JeffLuoo) · 2026-08-12T18:40:48Z

Hi Max, can you please review it? It doesn't generate any report yet, just set up things we need to benchmark observability. Max Smythe (@maxsmythe)

Max Smythe (maxsmythe)

This looks awesome! Couple questions that may just need clarificatoin.

Max Smythe (maxsmythe) · 2026-08-13T01:47:04Z

+`cycles/sec ≈ users / (1.0s + avg_wait_time)`. Change the number of users. Do
+not change the wait time.
+
+| # | Scenario | Duration | Setup |


Note that Python doesn't play well with gRPC when the number of users gets high enough, causing seconds worth of latency and throttling QPS. This is why the switch to boomer occurred.

Sleep currently uses Python worker. Might want to switch to glutton user to avoid this.

Yeah, I can update it to glutton in a follow-up that adds the ladder to the config.

Max Smythe (maxsmythe) · 2026-08-13T03:43:22Z

+
+## 2. Open items
+
+- Add the ladder to [automation/tests.yaml](automation/tests.yaml). Write the


The ladder seems like something that locust/runner.py should execute in some way.

The non-linear traffic pattern is something that is ideal to provide as a custom shape (locust/shapes)

Changing trace probability dynamically is an interesting wrinkle.

The Boomer worker does dynamically fetch config from master:

substrate/internal/benchmarking/boomer/dynconfig/dynconfig.go

Lines 89 to 109 in febeb7a

// Fetch GETs `url` and merges any returned fields into `current`. Returns

// the merged Config (or current unchanged on a soft no-op response).

func Fetch(ctx context.Context, url string, current Config) (Config, error) {

req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)

if err != nil {

return current, err

}

resp, err := http.DefaultClient.Do(req)

if err != nil {

return current, fmt.Errorf("GET %s: %w", url, err)

}

defer resp.Body.Close()

if resp.StatusCode != http.StatusOK {

return current, fmt.Errorf("GET %s: status %d", url, resp.StatusCode)

}

var p payload

if err := json.NewDecoder(resp.Body).Decode(&p); err != nil {

return current, fmt.Errorf("decode %s: %w", url, err)

}

return p.merge(current), nil

}

So it is possible. Reloads occur whenever worker receives a spawn event from locust, though that may not always be an intuitive trigger.

Agree, and this is already the planned follow-up. This PR focus on getting manifest ready.

Makes sense, should have said this was an informational comment rather than a blocker.

Krisztian F (krisztianfekete) · 2026-08-13T11:22:16Z

+        traces/count:
+          receivers: [otlp]
+          processors: [transform/svc]
+          exporters: [count]
+        traces/forward:
+          receivers: [otlp]
+          exporters: [otlp/managed]
+        metrics/count:
+          receivers: [otlp]
+          processors: [transform/svc]
+          exporters: [count]
+        metrics/forward:
+          receivers: [otlp]
+          exporters: [otlp/managed]
+        metrics/counts:
+          receivers: [count]
+          processors: [deltatocumulative]
+          exporters: [prometheus]


Can we add memory_limiter, and configure GOMEMLIMIT?

This is just a tee needed by GKE Managed OTel collector.

This tee collector is not needed for kind cluster and it should only process telemetry data and stream it to the GKE Managed collector, which means this is not a "real" collector. I am open to add memory_limiter and configure GOMEMLIMIT but just think it's not necessary.

I these are still beneficial for any perf/loadtesting setups to ensure reliability, but we can add them as a follow-up.

I will keep this comment open to allow me track it.

Krisztian F (krisztianfekete) · 2026-08-13T11:24:24Z

+      # long run out of Cloud Monitoring, but it makes the resource values of
+      # the managed collector those of an idle collector.
+      otlp/managed:
+        endpoint: opentelemetry-collector.gke-managed-otel.svc.cluster.local:4317


Same as in deploy.sh, can we make this work with local clusters as well?

Maybe we should also make this use the centralized OTel config introduced in #746?

We don't need the meter.yaml for kind cluster because the otel collector is customized. Instead, I think I can have a follow-up to update the manifest of the otel collector deployed to the kind cluster and integrate it with the benchmark.

I can do it in a follow-up.

Part of agent-substrate#563. The suite measured how fast substrate is. Nothing measured how much telemetry it produces, or what that telemetry costs the OTel collector. This change adds the measurement, and builds it from the parts that the suite already has. The managed collector sends its self-metrics to Cloud Monitoring only, and gives no data for each source. Thus benchmarking/telemetry/meter.yaml adds a second collector that counts the spans and the datapoints by service.name. The meter is a tee: it counts the telemetry and then sends it on to the managed collector, thus one run gives the volume for each service and the cost of the load together. Contents: - telemetry/meter.yaml and telemetry/README.md: the instrument and the method. Install, measure, read, remove. - hack/install-ate.sh --otlp-endpoint: sends all control plane telemetry to the meter. Each component reads the endpoint from the ate-otel-config ConfigMap through envFrom, and ate-controller copies the value to the ateom worker pods, thus one patch is sufficient for all of them. The override runs after the bundle apply, because the bundle has its own copy of the ConfigMap and would replace an earlier patch with no error message. deploy_locust.sh forwards the flag. - workloads/deploy.sh --otlp-endpoint: the same for actor containers, which substrate configures separately. - monitoring.yaml: scrapes the meter and cAdvisor. - observability.md: the prerequisites and the scenario ladder. - docs/dev/best-practices/otel-collector.md: the volume model and the size guidance for Deployment or DaemonSet. This file already had a shorter form of the same model, which was incorrect after the ParentBased defaults from agent-substrate#711. Correction: the glutton ActorTemplate had no env. Substrate puts no OTLP configuration in an actor container, thus the exporter used the SDK default localhost:4317, no process listens on that address in the sandbox, and substrate discarded each actor span with no error message. The template now sets OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_RESOURCE_ATTRIBUTES. ActorTemplate.spec is immutable, thus the deploy function deletes the templates before it applies them. Findings: - With ParentBased, the caller makes the root decision. Locust always sends a traceparent, thus its --trace-probability is the root rate for benchmark traffic and the default in ateapi does not apply. - The metrics increase with the number of components, not with the load. The spans increase with the rate of the operations. - An idle worker pod sends zero datapoints. Lifecycle traffic starts the telemetry of ateom, not the number of actors. - atelet and ateom-gvisor send no spans of their own. They inherit the decision that ateapi makes. - Actor throughput becomes constant at the limit of the snapshot suspend and restore operations, not at a limit of the telemetry. At that load the collector used a small part of its memory limit but stayed at maxReplicas, thus throughput is the wrong axis for the size calculation. The number of pods in the cluster sets the limit. The guidance is qualitative. It gives the shape of each curve and tells the reader to measure the values for their own cluster. No result tables go in the repository, because a result becomes incorrect after a change to the sample rates or to the instruments. All prose is in Simplified Technical English (ASD-STE100).

Co-authored-by: Krisztian F <103492698+krisztianfekete@users.noreply.github.com>

Max Smythe (maxsmythe)

LGTM

The scenario ladder was a table in observability.md that an operator followed by hand. Each step was a separate run, thus each step also carried a deploy, a set of cold caches, and a new baseline for the reader to align. The sample-rate sweep needed a new run for each rate, because the rate reached the boomer worker one time, at its start. Add shapes/ladder_shape.py and put the ladder in tests.yaml. `--ladder users:duration[:trace_probability],...` makes the steps of a sweep one run. The shape holds each step for its duration and then goes to the next one, thus it owns the end of the run and runner.py forwards no `-t` for a ladder. A step that names a probability changes the sample rate as it starts: the value goes to the parsed options of the master, thus a Python user class reads it from the sampler of common.trace and a boomer worker reads it from /boomer-config at the spawn message that the step change makes. A headless master has no Flask app, thus it had no /boomer-config and runner.py gave boomer the values one time through --config-json. That is sufficient for a run that holds one set of values, and not for a ladder. Serve the same payload from a plain HTTP server on the loopback interface when --boomer-config-port is set, which runner.py sets for a ladder along with the same port as boomer's --master-web-port. The four tests run on glutton, and thus on boomer. maxsmythe called this out on agent-substrate#749: Python and gRPC hold each other at a high number of users, so a ladder on SleepUser would put the latency of the load generator in the measurement. The user sweep stops at 15 users, and not at the 30 of the earlier table, because the pool of the run is 10 workers; a step above that measures the queue of the scheduler. The idle floor sends no request on purpose, thus locust writes no measurement row for it and runner.py counted the run as a failure. Add --allow-empty-stats for that one case, rather than accepting an empty result for every test. Follow-up to agent-substrate#749.

Follow-up to #749. One of three; the other two are independent of this one. This closes the review thread that stayed open on that PR: krisztianfekete asked for `memory_limiter` and `GOMEMLIMIT` on the meter, and I kept it open to track.

The scenario ladder was a table in observability.md that an operator followed by hand. Each step was a separate run, thus each step also carried a deploy, a set of cold caches, and a new baseline for the reader to align. The sample-rate sweep needed a new run for each rate, because the rate reached the boomer worker one time, at its start. Add shapes/ladder_shape.py and put the ladder in tests.yaml. `--ladder users:duration[:trace_probability],...` makes the steps of a sweep one run. The shape holds each step for its duration and then goes to the next one, thus it owns the end of the run and runner.py forwards no `-t` for a ladder. A step that names a probability changes the sample rate as it starts: the value goes to the parsed options of the master, thus a Python user class reads it from the sampler of common.trace and a boomer worker reads it from /boomer-config at the spawn message that the step change makes. A headless master has no Flask app, thus it had no /boomer-config and runner.py gave boomer the values one time through --config-json. That is sufficient for a run that holds one set of values, and not for a ladder. Serve the same payload from a plain HTTP server on the loopback interface when --boomer-config-port is set, which runner.py sets for a ladder along with the same port as boomer's --master-web-port. The four tests run on glutton, and thus on boomer. maxsmythe called this out on agent-substrate#749: Python and gRPC hold each other at a high number of users, so a ladder on SleepUser would put the latency of the load generator in the measurement. The user sweep stops at 15 users, and not at the 30 of the earlier table, because the pool of the run is 10 workers; a step above that measures the queue of the scheduler. The idle floor sends no request on purpose, thus locust writes no measurement row for it and runner.py counted the run as a failure. Add --allow-empty-stats for that one case, rather than accepting an empty result for every test. Follow-up to agent-substrate#749.

Follow-up to #749, for comment of adding the ladder design in benchmarking for observability.

The volume measurement worked on GKE only. There the managed collector cannot hold a connector, thus benchmarking/telemetry/meter.yaml puts a second collector in front of it and counts the data as it goes through. On kind the collector is ours, thus it can do the count itself and the cluster needs no meter and no extra hop. Add the count connector to the kind collector. It now gives the same two counts as the meter, substrate_spans_total{service_name} and substrate_datapoints_total{service_name}, on the Prometheus exporter port that it already opens. The connector resolves each key from the attributes of the item, then the scope, then the resource, thus service.name resolves with no transform in front of it. Drop the transform from the meter also. Give the collector its otelcol_* counters also. The default reader binds to localhost, thus nothing outside the pod could read it. Bind it to each interface and open port 8888. The counters are the positive control of a measurement: without them a service that reports zero volume and data that never arrived look the same. One scrape job reads both ports. The prometheus.io/scrape annotation can name one port only, and port 8889 re-exports each metric that the pipelines push. The pods of ateapi, atelet, and atenet-router already carry that annotation on 9090, thus an annotation on the collector would bring each of their series in a second time and sum() would give double. The job keeps the substrate_* and otelcol_* names only. The actors need no change. benchmarking/workloads/deploy.sh reads the endpoint from the ate-otel-config ConfigMap, and the kind overlay of that ConfigMap already names this collector. Follow-up to agent-substrate#749.

The volume measurement in current benchmarking repo worked on GKE only. There the managed collector cannot hold a connector, thus benchmarking/telemetry/meter.yaml puts a second collector in front of it and counts the data as it goes through. This PR adds the support to kind cluster to count the telemetry data. Follow-up to #749. > It's a good idea to open an issue first for discussion. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR

Jeff Luo (JeffLuoo) requested review from Da Huang (git286) and Max Smythe (maxsmythe) August 4, 2026 22:20

Krisztian F (krisztianfekete) reviewed Aug 5, 2026

View reviewed changes

Jeff Luo (JeffLuoo) marked this pull request as draft August 12, 2026 15:04

Jeff Luo (JeffLuoo) force-pushed the obs-bench branch 10 times, most recently from 85fc01e to 0011ab3 Compare August 12, 2026 18:30

Jeff Luo (JeffLuoo) marked this pull request as ready for review August 12, 2026 18:35

Jeff Luo (JeffLuoo) force-pushed the obs-bench branch from 0011ab3 to b79b2ef Compare August 12, 2026 18:37

Jeff Luo (JeffLuoo) requested a review from Krisztian F (krisztianfekete) August 12, 2026 18:45

Max Smythe (maxsmythe) requested changes Aug 13, 2026

View reviewed changes

Krisztian F (krisztianfekete) reviewed Aug 13, 2026

View reviewed changes

Jeff Luo (JeffLuoo) force-pushed the obs-bench branch 3 times, most recently from ea2deda to 8b172ad Compare August 13, 2026 19:24

Jeff Luo (JeffLuoo) force-pushed the obs-bench branch from 8b172ad to 2661df9 Compare August 13, 2026 21:05

Jeff Luo (JeffLuoo) requested review from Krisztian F (krisztianfekete) and Max Smythe (maxsmythe) August 13, 2026 21:07

Krisztian F (krisztianfekete) reviewed Aug 14, 2026

View reviewed changes

Comment thread benchmarking/telemetry/README.md Resolved

Comment thread docs/dev/best-practices/otel-collector.md Resolved

Update benchmarking/telemetry/README.md

a37e2b5

Co-authored-by: Krisztian F <103492698+krisztianfekete@users.noreply.github.com>

Jeff Luo (JeffLuoo) requested a review from Krisztian F (krisztianfekete) August 14, 2026 13:40

Max Smythe (maxsmythe) approved these changes Aug 15, 2026

View reviewed changes

Da Huang (git286) approved these changes Aug 17, 2026

View reviewed changes

Da Huang (git286) merged commit 9bfc67e into agent-substrate:main Aug 17, 2026
11 checks passed

This was referenced Aug 19, 2026

benchmarking: hold the memory of the telemetry meter #1069

Merged

manifests: count telemetry by service on kind #1070

Merged

benchmarking: run the observability ladder as one test #1071

Merged

Max Smythe (maxsmythe) pushed a commit that referenced this pull request Aug 22, 2026

benchmarking: run the observability ladder as one test (#1071)

4bc650a

Follow-up to #749, for comment of adding the ladder design in benchmarking for observability.


		## 2. Open items

		- Add the ladder to [automation/tests.yaml](automation/tests.yaml). Write the

	// Fetch GETs `url` and merges any returned fields into `current`. Returns
	// the merged Config (or current unchanged on a soft no-op response).
	func Fetch(ctx context.Context, url string, current Config) (Config, error) {
	req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
	if err != nil {
	return current, err
	}
	resp, err := http.DefaultClient.Do(req)
	if err != nil {
	return current, fmt.Errorf("GET %s: %w", url, err)
	}
	defer resp.Body.Close()
	if resp.StatusCode != http.StatusOK {
	return current, fmt.Errorf("GET %s: status %d", url, resp.StatusCode)
	}
	var p payload
	if err := json.NewDecoder(resp.Body).Decode(&p); err != nil {
	return current, fmt.Errorf("decode %s: %w", url, err)
	}
	return p.merge(current), nil
	}

Conversation

Jeff Luo (JeffLuoo) commented Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Contents

Correction

How I verified this

Result

Traces

Follow-up

Uh oh!

Krisztian F (krisztianfekete) left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Jeff Luo (JeffLuoo) commented Aug 5, 2026

Uh oh!

Jeff Luo (JeffLuoo) commented Aug 12, 2026

Uh oh!

Max Smythe (maxsmythe) left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Jeff Luo (JeffLuoo) Aug 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Max Smythe (maxsmythe) left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Jeff Luo (JeffLuoo) commented Aug 4, 2026 •

edited

Loading

Jeff Luo (JeffLuoo) Aug 13, 2026 •

edited

Loading