[Krisztian F (krisztianfekete)]

What this does

ateapi writes a record every time an actor changes state. Until now those records only went to the pod's stdout, and nothing reads stdout. This sends the same records to a collector as OTLP log events.

Actor name and uid cannot be metric labels (too many values), and traces are sampled at 1%. So these records are the only way to answer "what state is this actor in, and since when".

Changes

• serverboot.InitLogging sets up a LoggerProvider, next to the existing tracer and meter ones.
• New internal/actorevent package builds the log records.
• ateapi emits at the two places that already write the stdout records.
• Two event names: ate.actor.state_changed and ate.actor.crashed.
• Both names are registered in docs/metrics/registry/events.yaml, so make verify checks them.
• kind gets a logs pipeline and a count connector. The e2e suite reads the counts back.
• Docs updated. otel-collector.md said substrate has no LoggerProvider, which is no longer true.

Opt-in

OTEL_LOGS_EXPORTER defaults to none. Only the kind overlay sets it to otlp. The base ConfigMap is untouched, so no deployed environment changes when this merges.

Notes on the design

• No slog bridge. Only two call sites emit these records, so emitting twice costs two lines. A bridge would also send every ateapi log over the wire, could not set the event name, and would loop, because SDK export errors are logged through slog.
• Batching processor, not the simple one. These records sit on the actor resume path. A processor that exports inside the emit call would add a blocking gRPC call there, so a slow collector would become control plane latency.
• Two event names, not one per state. ate.actor.state already says which transition happened. A crash gets its own name because it carries two extra attributes and a higher severity.
• Both copies are kept on purpose. No collector in this repo reads pod stdout, so nothing is duplicated today. kubectl logs keeps working. If a filelog agent is ever added, drop one of the two. The escape hatch is written down in docs/metrics/substrate.yaml.

Dependencies

Adds otel/log, otel/sdk/log and otlploggrpc, all pinned at v0.20.0. That is the release that matches the pinned otel v1.44.0. v0.21.0 would pull the core modules to v1.45.0, which this change does not need. The logs API has a
v1.47.0 release candidate upstream, so it is on its way to stable.

Testing

• Unit tests for the exporter resolver, the record builder, and both ateapi emit sites.

• The record builder test checks the attribute set matches what the event name declares, in both directions.

• The ateapi tests check the OTLP record carries the same attributes as the stdout record.

• Ran end to end on kind. Records arrive with the right event name, severity, attributes, and with trace context on the record's own fields rather than as attributes.

• Checked the off state too. With OTEL_LOGS_EXPORTER removed, the collector receives no log records and stdout is unchanged.

• Tests pass

• Appropriate changes to documentation are included in the PR

[Krisztian F (krisztianfekete)]

Jeff Luo (@JeffLuoo), can you please take a look at this when you have some free cycles?

[Jeff Luo (JeffLuoo)]

Sure!

[Jeff Luo (JeffLuoo)]

Can you first create a bug to track the work and general discussion?

[Krisztian F (krisztianfekete)]

Can you first create a bug to track the work and general discussion?

Sure, wanted to open one for related topics, but I think we can keep these reviewed in parallel as this PR

• adds no new records as ateapi has written both to stdout for a while now, so this is only about the OTLP support, and
• is off everywhere by default as only the kind overlay sets OTEL_LOGS_EXPORTER

Opened the discussion here: #1683

[Jeff Luo (JeffLuoo)]

log.String(a.Key, a.Value.String()) is lossless today, because ActorLogAttrs and FailureLogAttrs build only slog.String. The Keys test pins the key names, not the value kinds. So the first slog.Int or slog.Duration attribute gives a number on stdout and a string over OTLP, and no test fails. The package doc says the two copies cannot drift, and this is the one way they can. A type switch, or a test that asserts every attribute is string-kind, would help it.

[Krisztian F (krisztianfekete)]

Thanks, good catch, this is now addressed.

[Jeff Luo (JeffLuoo)]

nit - "actor lifecycle events" is what we export today but we expect to have more events coming from the system. Let's make it general like "ate events" or something similar.

[Krisztian F (krisztianfekete)]

Done.

[Jeff Luo (JeffLuoo)]

This reads the bare otlp receiver with no filter, and the connector at line 88 defaults ate.actor.state to unknown. The same collector doc says the logs pipeline also serves user workloads, so a demo actor that emits OTLP logs lands in substrate_actor_state_changes{ate_actor_state="unknown"}. The e2e is unaffected, because it reads named states. Worth a filter on the scope, or a description at line 84 that says what is actually counted. If we want to just track state change count, add a filter here.

[Krisztian F (krisztianfekete)]

This is now fixed by just dropping default_value on ate.actor.state. The connector drops items missing a key that has no default, so we don't even need a processor here. A real event with state unknown still has the key so that still counts.

[Jeff Luo (JeffLuoo)]

Could we accept console here as a third value? Just easier to test change.. Or are you intentional not adding it?

[Krisztian F (krisztianfekete)]

I intentionally left it out. We already write the same record to stdout so the console version would print the same thing in slightly different format which I think is more confusing than useful here.

As per seeing the OTLP format, the collector's debug exporter already dumps the full record, so unit tests can assert that directly.

[Jeff Luo (JeffLuoo)]

As per seeing the OTLP format, the collector's debug exporter already dumps the full record, so unit tests can assert that directly.

I want to break it down into 2 parts:

1. Instrumentation using OTel SDK for logs / events - Yes. And we should make it freely configurable, including options to write logs to console (stdout).

2. Collection part: We shouldn't assume a collector will be deployed. Users can use any collection agent.

Hence my point is we shouldn't limit the configuration of OTel SDK, and provide users guidance to configure.

[Krisztian F (krisztianfekete)]

stdout already has the same information in ate.* format in JSON that is readable and parsable by any agent/collector. Adding console will result in the exact same data, duplicated in a slightly different format.

[Jeff Luo (JeffLuoo)]

Thanks, I agree that a console exporter is not needed here: the existing structured stdout record remains available to kubectl logs and to a future filelog collector, while OTLP provides the schema-specific event form.

I do want to make one long-term invariant explicit: one lifecycle event should be constructed once, then projected to stdout and OTLP, rather than maintained as two parallel emissions at each call site. Reusing []slog.Attr is a good start, but slog.LogAttrs(...) and actorevent.Emit(...) still leave separate body/severity/emission paths that can drift as the event evolves.

Would it make sense to have actorevent own a single higher-level emission helper or canonical event record and derive both outputs from it? This is not a blocker for this PR, but I would like us to capture the invariant in the design or a follow-up issue so it does not get lost.

[Krisztian F (krisztianfekete)]

Sure, opened it here, can pick it up as a follow-up once this one is merged: #1744