Repository navigation
Actor instrumented metrics might merge and inflate #761
Description
Any workload that exports OTel metrics loses per-actor identity when it moves to Substrate. Counters come out wrong, and nothing errors.
Two things cause it.
Everything computed at startup is shared. The golden snapshot is taken after init, so an OTel Resource built in main() is identical for every actor of the template. The usual escapes don't work either: os.Hostname() is hardcoded to runsc for every sandbox (cmd/atelet/oci.go:271), and ActorTemplate doesn't support valueFrom.fieldRef, so you can't inject a pod name (docs/api-guide.md:101).
Collector enrichment attributes to the worker pod. Actor egress is masqueraded behind the pod IP in both ateom runtimes (cmd/ateom-microvm/net.go:22), so k8sattributes stamps the host worker's k8s.pod.name / k8s.node.name. That merges every co-located actor into one series, and changes an actor's series identity every time it migrates. Every actor also shares one constant interior IP (169.254.17.2), so no IP-based processor can tell them apart even in principle.
Note that push doesn't get us out of this, it's what gets us into it. Actors can't be scraped, and the pull model is what hands you an instance label for free, derived from the target address rather than from anything the workload says. With OTLP push, identity has to come from the payload, and the payload is the part that's frozen.
The result is one series with many writers. OTLP counters default to cumulative, so the merged series goes up and down, rate()/increase() treat every decrease as a reset, and totals inflate. Gauges show a random actor. The dashboards look plausible, which is the worst part and the pod/node labels make it read like ordinary pod churn, so you debug the wrong thing.
Adding actor_uid as a metric label isn't the fix; it's unbounded and churns on every actor create.
Things I think we need to decide:
- what identifies an actor's metrics on the wire vs. what survives into storage
- whether Substrate stamps it out-of-process (the identity dir seems like the natural place, since it's outside the snapshot) or we leave it to actor authors
- interim guidance for actor authors either way
Related: #503 (telemetry continuity), #450 (lifecycle hooks). PR #750's demo hits this: its actorName() returns runsc for every actor.
Activity
git286 commentedon Aug 5, 2026
cc Jeff Luo (@JeffLuoo)
krisztianfekete commentedon Aug 6, 2026
Thanks Da Huang (@git286), this is something I've been tracking for a while but never got around writing a proposal.
Couple of notes/questions:
What do you mean by co-located actors? One worker can only host one actor today, right?
We don't have this in our pipeline config yet, but it's actually even worse as
pod_associationis connection-bsed by default, and an actor is not a pod, so no k8s.* attribute can describe it properly.The bigger problem here is that since actor restores from the same golden snapshot, every actor gets the golden's accumulated values and the regular reset detection won't work either.
But we cannot use this for microvm, right?
To me, using something like
PostResumefrom #450 seems more reasonable on the long run if we want something in-process.Another solution I've been thinking about is pointing actors at a fixed OTLP endpoint on the worker, overwrite telemetry identity there then relay to a real collector.
I've been thinking about setting aggregation temporality to delta, that's what's recommended for Lambda-like workloads anyway. It will fix counters/histograms, but not gauges. For that we'd need a something more sophisticated on the metrics front likely via #450.
I think the best interim solution is using logs/traces for anything that requires identity, and set delta temporality to fix the trivial metrics.
Let me know what you think!
AiWatson343 commentedon Aug 6, 2026
This highlights a fundamental identity boundary issue with workload migration. The important distinction is that actor identity cannot be derived from runtime-local properties (hostname, IP, worker pod metadata) because those are either shared or intentionally change during scheduling/migration.
I agree that adding actor_uid directly as a metric label is not a sustainable solution due to cardinality and churn. We likely need a two-layer model:
On-wire identity: a stable actor/workload identity that Substrate injects before telemetry leaves the sandbox.
Storage identity: a controlled set of attributes suitable for aggregation, retention, and querying without creating unbounded series.
The identity directory approach seems promising because it exists outside the snapshot boundary and can provide runtime-specific metadata without depending on frozen startup state. Substrate could potentially enrich OTLP payloads out-of-process, similar to how collectors add infrastructure metadata today, but using actor identity rather than worker identity.
For interim guidance, actor authors should avoid building OTel resources from host-derived values (hostname, pod name, IP) and should treat those as infrastructure context only. They need a documented Substrate-provided identity mechanism until the platform owns this enrichment path.
This also seems closely connected to lifecycle hooks: actor creation, migration, restore, and termination events are natural points where identity continuity can be preserved and validated. The main goal should be that an actor’s telemetry follows the actor lifecycle without inheriting the identity of the worker that happens to run it.
git286 commentedon Aug 6, 2026
Sorry, I mean the actors on the same pod sequentially, not concurrently
For GKE managed OTel, we have the auto attribution config on by default
Yes, golden snapshot is causing a big issue here
You're right that a Unix socket doesn't work for microvm. The identity bind mount isn't available in the guest at all. But a network endpoint is. Both runtimes already wire the actor to the same worker-side address (169.254.17.1), so micro-VM comes along for free.
In general, I think solutions requires actor side code changes is less ideal as it will add more friction for workloads to migrate to substrate. (Probably ok as interim solution)
Yeah, I was thinking something similar (still need more thoughts though): the high level idea is to relay the metrics to ateom (and then atelet), which knows the actor identity and can do some processing before sending the data to the centralized otel collector
Agreed. For M2, we probably won't land the proper relay solution and I am thinking of the same interim solution by recommending the actor developers to follow some best practices on their side like setting delta etc.
krisztianfekete commentedon Aug 7, 2026
Da Huang (@git286), do you mind creating a short design doc so we can finalise the approach? Happy to do it if you're busy with other stuff, just let me know.
git286 commentedon Aug 7, 2026
Sure, I will try to have some design early next week. Will let you know if I don't get chance to work on it.
otlprelay: scope the relay to ateom, and fix two socket path bugs