Skip to content

OTLP exporters hardcode WithInsecure(), so the SDK's TLS config never applies #741

Open

Description

@git286

Related: #563

serverboot.go:141 builds the trace exporter with an unconditional otlptracegrpc.WithInsecure(), and serverboot.go:182 does the same for metrics. internal/benchmarking/boomer/trace/trace.go:89 repeats it.

The OTel SDK reads env config first and applies explicit options after, so that one option overrides everything the exporter already knows how to do:

cfg = ApplyGRPCEnvConfigs(cfg)   // OTEL_EXPORTER_OTLP_CERTIFICATE, CLIENT_CERTIFICATE,
for _, opt := range opts {       // CLIENT_KEY, INSECURE, and http:// vs https:// inference
    cfg = opt.ApplyGRPCOption(cfg)   // ...all discarded here
}

So OTEL_EXPORTER_OTLP_ENDPOINT=https://collector:4317 silently exports plaintext. No error, no warning. Already noted in otel-collector.md:458 as a known constraint.

The comment justifies it with GKE managed OTel, which is fair — that collector genuinely has no TLS receiver you can configure. But serverboot boots every binary in every environment, and we document a self-managed DaemonSet path (otel-collector.md:64) where TLS and mTLS work fine. A GKE Preview limitation shouldn't be compiled in for everyone.

Fix

Delete the WithInsecure() calls and let scheme inference plus the standard env vars do their job. Log the resolved transport next to the existing "Tracing initialized" line so this is visible without reading source. Then fix the "TLS is not supported" text in otel-collector.md:458 and tracing.md:49.

Not proposing we enable TLS by default or wire pod certs into the exporter — managed OTel can't accept either. This is just about not blocking people whose collector can.

Compat

No manifest changes needed. Every Go-SDK endpoint we ship already has an explicit http:// scheme (the four in ate-install/, the five under kind/), which infers plaintext. Unchanged on GKE and Kind. The scheme-less AGENTGATEWAY_OTLP_ADDRESS is agentgateway's, not ours.

One thing to check first: with no OTEL_EXPORTER_OTLP_ENDPOINT set, the SDK defaults to localhost:4317 with Insecure: false, so those processes would start attempting TLS. Need to confirm before merging:

  • do ateom-gvisor / ateom-microvm inherit the env from atelet? They call InitTracing but have no manifest of their own
  • atecontroller — ate-controller.yaml:96 sets the endpoint but I don't see it in the InitTracing call sites
  • local go run and any test that boots a real exporter

Failure mode is background export errors rather than a crash, but worth sorting out rather than finding in prod.

Activity

  1. added this to the M2 milestone on Aug 4, 2026
  2. git286 commented on Aug 4, 2026

    @git286
    CollaboratorAuthor
  3. JeffLuoo commented on Aug 4, 2026

    @JeffLuoo
    Collaborator

    To further break it down, there are two paths involved:

    • For Telemetry transmission between ate system components and OpenTelemetry Collector(s)
    • For Telemetry transmission between actor (the workload container) and OpenTelemetry Collector(s)
  4. JeffLuoo commented on Aug 4, 2026

    @JeffLuoo
    Collaborator

    Dropped some open questions:

    • Substrate has two identity systems: pod certs (podidentity.podcert.ate.dev, SPIFFE URI SAN) and a separate actor identity (cmd/ateapi/internal/actoridentity/, JWT + X.509 from internal/localca, ActorIdentity extension at OID 1.3.6.1.4.1.11129.2.12.2). Is the pod the intended trust boundary for telemetry, or the actor?
  5. krisztianfekete commented on Aug 5, 2026

    @krisztianfekete
    Collaborator

    Thanks for opening this!

    Couple of notes:

    • There's also benchmarking/locust/manifests/locust.yaml, where the endpoint doesn't have scheme. That will become and empty endpoint, so we have to fix that as well.
    • Re. Jeff Luo (@JeffLuoo)'s points: ateom workers get the endpoint propagated by atecontroller, not inherited from atelet, and both GKE and
      kind set it. atecontroller never inits an exporter, its only using ENV vars, so the only real regression is local dev against a plaintext localhost collector without ENV vars I think.
    • Given that, I'd guard instead and keep WithInsecure() only when no OTEL_EXPORTER_OTLP_* ENV var is set.
    • Regarding the log line, I'd log the raw ENV variables, don't think SDK can see the transport
  6. JeffLuoo commented on Aug 12, 2026

    @JeffLuoo
    Collaborator

    TLS for actor will depend on the outcome of #853. If we proceed with the proposal, the identity will be atelet that is responsible for relaying the telemetry data to otel collecotr.

  7. added
    kind/featureAn enhancement / feature request or implementation
    and removed
    kind/bugSomething isn't working / bugfixes
    on Aug 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

area/observabilitykind/featureAn enhancement / feature request or implementationprio/P0Highest priority / required for next milestone

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions