demos: add actor telemetry continuity reference demo (#503) by sanskar-singh-2403 · Pull Request #750 · agent-substrate/substrate

Sanskar Singh Bhardwaj (sanskar-singh-2403) · 2026-08-05T04:33:00Z

Add a reference demo for actor telemetry continuity across suspend/resume, the
follow-up work requested in #503.

The demo is a small Go HTTP server that emits OpenTelemetry metrics and trace
spans over OTLP on every request. It reproduces the #503 loss: an OTel push
exporter batches telemetry in memory and flushes on a timer, so when the push
interval (default 60s) is longer than the active window before idle suspension,
telemetry is still buffered in the SDK when the actor is suspended. With
snapshotsConfig.onPause: Data (the mode the template uses) Substrate drops
process RAM on pause, so that buffered telemetry is lost.

The mitigation is isolated behind a flushTelemetry(ctx) seam that ForceFlushes
both the MeterProvider and TracerProvider. Today it is called from the
self-suspend path via an idle watcher; when the PreSuspend hook (#450) lands,
flushTelemetry does not change, only its call-site moves into the PreSuspend
handler. This keeps #450 a call-site move rather than a rewrite.

The ActorTemplate sets OTEL_EXPORTER_OTLP_ENDPOINT to the in-cluster
collector, since the SDK default (localhost:4317) has nothing listening inside
the actor sandbox netns.

Includes install-demo-actor-telemetry.sh wired into install-ate.sh, a
rendered-and-validated template, unit tests for the idle-flush watcher, and a
README documenting the reproduce/verify flow against Jaeger and Prometheus on
Kind.

Scope

This covers the "reference demo for actor telemetry continuity" action item in
#503 only. The docs/observability.md update is tracked separately, and the
PreSuspend/PostResume alignment is #450.

Part of #503

Testing

gofmt, go vet, go build, and go test ./demos/actor-telemetry/ all pass
(4/4 unit tests on the idle-flush watcher).
Template renders via the install script's sed substitution and parses as
valid Kubernetes manifests.
End-to-end suspend/resume verification against a live Kind cluster with
ate-system is described in the README.
Tests pass
Appropriate changes to documentation are included in the PR

Da Huang (git286) · 2026-08-05T19:12:27Z

+	return res, nil
+}
+
+func actorName() string {


ATE_ACTOR_NAME isn't set anywhere in the repo, and os.Hostname() returns "runsc" for every actor — it's hardcoded in the OCI spec (cmd/atelet/oci.go:271) and applied via the UTS namespace. So this always returns runsc.

The documented way is /run/ate/actor-id (docs/api-guide.md:108), with a working example in internal/e2e/fixtures/probe/main.go:32. api-guide.md:111 also warns specifically against the env-var approach, since env is frozen in the golden snapshot.

Good catch, confirmed. os.Hostname() returns runsc for every actor (hardcoded at cmd/atelet/oci.go:268) and ATE_ACTOR_NAME is set nowhere, so this always returned runsc. Switched to reading /run/ate/actor-id fresh per the guidance in docs/api-guide.md, mirroring internal/e2e/fixtures/probe/main.go.

Da Huang (git286) · 2026-08-05T19:13:18Z

+	ctx := context.Background()
+	slog.SetDefault(slog.New(slog.NewJSONHandler(os.Stdout, nil)))
+
+	res, err := newResource(ctx)


This resource is built in main(), so it's captured by the golden snapshot and identical for every actor of the template: even reading /run/ate/actor-id here would give you the golden actor's name.

Right, building it in main() freezes it into the golden snapshot. Moved actor identity out of the resource entirely and now read it fresh in the request handler, so it reflects the resumed actor rather than the golden one. Updated the test to cover the fresh-read and the missing-file fallback.

Da Huang (git286) · 2026-08-05T19:18:51Z

+	requests, err := meter.Int64Counter(
+		"demo.actor.requests",
+		metric.WithDescription("Requests handled by the actor-telemetry demo, by actor."),
+	)


I just filed #761 : the counter might have the merge and conflate problem when actors run on different worker pods during its lifecycle.

got it that is the merge/conflate concern across worker pods and is orthogonal to this demo's fix; happy to track the counter-aggregation semantics there.

Da Huang (git286) · 2026-08-05T19:19:09Z

+		ctx, span := tracer.Start(r.Context(), "handle-request")
+		defer span.End()
+
+		requests.Add(ctx, 1, metric.WithAttributes(


ate.dev/actor_name on the metric datapoint is unbounded cardinality. The north star of this project is to have billions of actors.

On the span below it's fine, since a span is an event rather than a series.

For a reference demo I'd keep actor identity on the span and use template name on the metric.

Agreed, per-actor labels on the metric are unbounded cardinality at billions of actors. Changed the counter to label with the bounded ate.dev/template and kept ate.dev/actor_name on the span only, since a span is an event rather than a series. README updated to match.

Krisztian F (krisztianfekete) · 2026-08-06T12:22:54Z

+func getCurrentIP() string {
+	addrs, err := net.InterfaceAddrs()
+	if err != nil {
+		return "x.x.x.x"
+	}
+	for _, addr := range addrs {
+		if ipnet, ok := addr.(*net.IPNet); ok && !ipnet.IP.IsLoopback() {
+			if ipnet.IP.To4() != nil {
+				return ipnet.IP.String()
+			}
+		}
+	}
+	return "y.y.y.y"
+}


This is the os.Hostname() problem again. I'd either drop it, or keep it and relabel it to something like fake-static IP.

Removed getCurrentIP entirely. Per #761 every actor egresses behind one constant interior IP inside the gVisor netns, so it could never identify the worker pod. The response now reports the actor name (read fresh from the identity file) instead.

Krisztian F (krisztianfekete) · 2026-08-06T12:23:25Z

+   ```bash
+   curl -X POST -H "Host: my-telemetry-1.demo.actors.resources.substrate.ate.dev" http://localhost:8000
+   ```
+   Each response reports the worker pod that served it. The metric


This is not correct, see https://github.com/agent-substrate/substrate/pull/750/changes#r3728711006

Fixed. Reworded to "reports the actor it was handled for" now that the IP is gone.

Krisztian F (krisztianfekete) · 2026-08-06T12:27:23Z

+		sdktrace.WithBatcher(exp, sdktrace.WithBatchTimeout(*pushInterval)),
+	)
+	otel.SetTracerProvider(tp)
+	otel.SetTextMapPropagator(propagation.TraceContext{})


You set the propagator but never extract, so tracer always starts a root span and the actor's traces never join the router's. See #427

You can also use otelhttp.NewHandler to do avoid extraction and setting attrs manually.

Good catch. Wrapped the mux in otelhttp.NewHandler(mux, "/") so the incoming W3C trace context is extracted and the per-request span joins the router's trace instead of starting a detached root. Follows the same pattern as the atenet router and glutton.

Krisztian F (krisztianfekete) · 2026-08-06T12:28:54Z

+		// Metric: label with the bounded template name only. Per-actor labels
+		// are unbounded cardinality at this project's scale (see templateName).
+		requests.Add(ctx, 1, metric.WithAttributes(
+			attribute.String("ate.dev/template", templateName),


ate.dev/ is the k8s label and what we log to sdtout. Span and metric attributes use the ate.* namespace. So these should be ate.actor.template.name here and ate.actor.name below.

Please also update this in README.

Done. Switched to the dotted ate.* namespace: ate.actor.template.name on the metric and ate.actor.name on the span, matching internal/ateattr. Updated the README to match.

Krisztian F (krisztianfekete) · 2026-08-06T12:32:05Z

+// OTLP every push-interval. The long interval is the whole point: it is what
+// leaves telemetry buffered when a suspend arrives before the timer fires.
+func initMeterProvider(ctx context.Context, res *resource.Resource) (*sdkmetric.MeterProvider, error) {
+	exp, err := otlpmetricgrpc.New(ctx, otlpmetricgrpc.WithInsecure())


Please don't hardcode WithInsecure, see #741

Dropped the hardcoded WithInsecure() from both exporters. The demo endpoint is plaintext http://, which the SDK already infers from the scheme, so nothing regresses while the SDK's env-driven TLS config is now respected.

Krisztian F (krisztianfekete) · 2026-08-06T12:32:55Z

+	meter := mp.Meter(serviceName)
+	requests, err := meter.Int64Counter(
+		"demo.actor.requests",
+		metric.WithDescription("Requests handled by the actor-telemetry demo, by actor."),


Suggested change

metric.WithDescription("Requests handled by the actor-telemetry demo, by actor."),

metric.WithDescription("Requests handled by the actor-telemetry demo, by template."),

Krisztian F (krisztianfekete) · 2026-08-06T12:33:56Z

+	}
+
+	meter := mp.Meter(serviceName)
+	requests, err := meter.Int64Counter(


Nit: please add metric.WithUnit("{request}")

Krisztian F (krisztianfekete) · 2026-08-06T12:35:59Z

+    # listening, so telemetry would be flushed into the void. This mirrors the
+    # endpoint the ate-system components use on Kind (see
+    # manifests/ate-install/kind/kustomization.yaml).
+    - name: OTEL_EXPORTER_OTLP_ENDPOINT


Da Huang (@git286), I think it's a good opportunity to set delta temporarily here as well. See my reasoning in #761 (comment)

Set OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=delta in the template. Agreed it fits this suspend/resume workload: with cumulative, a restore from the golden snapshot carries the golden actor's accumulated value and reset detection cannot see the restart, so delta avoids the inflate/merge. Tracking the broader identity fix in #761.

Krisztian F (krisztianfekete) · 2026-08-06T12:36:30Z

+	res, err := resource.New(ctx,
+		resource.WithTelemetrySDK(),
+		resource.WithSchemaURL(semconv.SchemaURL),
+		resource.WithAttributes(


Please add service.instance.id

Added semconv.ServiceInstanceID, generated once and shared by the meter and tracer resources, mirroring internal/serverboot.

…#503) Add a reference demo for actor telemetry continuity across suspend/resume, the follow-up work requested in agent-substrate#503. The demo is a small Go HTTP server that emits OpenTelemetry metrics and trace spans over OTLP on every request. It reproduces the agent-substrate#503 loss: an OTel push exporter batches telemetry in memory and flushes on a timer, so when the push interval (default 60s) is longer than the active window before idle suspension, telemetry is still buffered in the SDK when the actor is suspended. With snapshotsConfig.onPause: Data (the mode the template uses) Substrate drops process RAM on pause, so that buffered telemetry is lost. The mitigation is isolated behind a flushTelemetry(ctx) seam that ForceFlushes both the MeterProvider and TracerProvider. Today it is called from the self-suspend path via an idle watcher; when the PreSuspend hook (agent-substrate#450) lands, flushTelemetry does not change, only its call-site moves into the PreSuspend handler. This keeps agent-substrate#450 a call-site move rather than a rewrite. The ActorTemplate sets OTEL_EXPORTER_OTLP_ENDPOINT to the in-cluster collector, since the SDK default (localhost:4317) has nothing listening inside the actor sandbox netns. Includes install-demo-actor-telemetry.sh wired into install-ate.sh, a rendered-and-validated template, unit tests for the idle-flush watcher, and a README documenting the reproduce/verify flow against Jaeger and Prometheus on Kind.

Sanskar Singh Bhardwaj (sanskar-singh-2403) mentioned this pull request Aug 5, 2026

OpenTelemetry Application Telemetry Data Loss and Delivery Delays Across Actor Lifecycle Events #503

Open

3 tasks

Da Huang (git286) mentioned this pull request Aug 5, 2026

Actor instrumented metrics might merge and inflate #761

Open

Da Huang (git286) reviewed Aug 5, 2026

View reviewed changes

Sanskar Singh Bhardwaj (sanskar-singh-2403) force-pushed the demo/actor-telemetry-503 branch from 521a7f0 to 5f87ad3 Compare August 6, 2026 11:23

Krisztian F (krisztianfekete) reviewed Aug 6, 2026

View reviewed changes

Sanskar Singh Bhardwaj (sanskar-singh-2403) force-pushed the demo/actor-telemetry-503 branch from 5f87ad3 to 46a8ac3 Compare August 6, 2026 17:26

Krisztian F (krisztianfekete) mentioned this pull request Aug 18, 2026

e2e: nothing catches actor metric continuity across suspend/resume #1036

Open

Bowei Du (bowei) added kind/feature An enhancement / feature request or implementation area/demos area/observability labels Sep 17, 2026

	metric.WithDescription("Requests handled by the actor-telemetry demo, by actor."),
	metric.WithDescription("Requests handled by the actor-telemetry demo, by template."),

Conversation

Sanskar Singh Bhardwaj (sanskar-singh-2403) commented Aug 5, 2026

Scope

Testing

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants