Skip to content

Support OTLP multi-endpoint fan-out (GH_AW_OTLP_ENDPOINTS) #7444

Closed

Description

@lpcox

Context

Reported in github/gh-aw#38901: The MCP gateway exports OpenTelemetry spans only to the single primary OTLP endpoint (OTEL_EXPORTER_OTLP_ENDPOINT). When gh-aw configures multiple OTLP backends via GH_AW_OTLP_ENDPOINTS, non-primary backends receive the gh-aw lifecycle spans but none of the gateway spans (gateway.request, mcp.tool_call, gateway.backend.execute).

Current Behavior

  • Gateway reads OTEL_EXPORTER_OTLP_ENDPOINT (single endpoint) and initializes one OTLP exporter
  • GH_AW_OTLP_ENDPOINTS is not passed into the gateway container
  • Non-primary backends get incomplete traces (0 gateway spans)

Desired Behavior

The gateway should support multi-endpoint OTLP export so all configured backends receive complete traces including gateway/tool-call spans.

Proposed Approach

  1. Accept GH_AW_OTLP_ENDPOINTS (comma-separated list of endpoint URLs) as a new env var
  2. When set, create one OTLP exporter per endpoint and fan out span exports to all
  3. Per-endpoint failure should not block export to other endpoints (partial-failure tolerance)
  4. Fall back to existing single-endpoint behavior when GH_AW_OTLP_ENDPOINTS is not set
  5. Each endpoint may have its own headers (need a format for per-endpoint headers, or use a shared OTEL_EXPORTER_OTLP_HEADERS for all)

Implementation Notes

  • The tracing setup lives in internal/tracing/ (provider initialization, OTLP HTTP exporter)
  • Current single-exporter setup: internal/tracing/provider.go
  • The --otlp-endpoint flag and OTEL_EXPORTER_OTLP_ENDPOINT env var remain the primary/fallback
  • Fan-out exporter could wrap multiple otlptracehttp exporters behind a composite SpanExporter interface

References

Activity

  1. github-actions commented on Jun 13, 2026

    @github-actions
    Contributor

    🤖 This is an automated response from Repo Assist.

    The comma-separated multi-endpoint fan-out (GH_AW_OTLP_ENDPOINTS) is already implemented in the codebase. Here's a summary of what's there and what remains:

    Already implemented

    • resolveExtraEndpoints() in internal/tracing/config_resolver.go reads GH_AW_OTLP_ENDPOINTS and splits by comma
    • newFanoutExporter() in internal/tracing/fanout.go wraps multiple sdktrace.SpanExporters with partial-failure tolerance
    • InitProvider uses GH_AW_OTLP_ENDPOINTS (takes precedence over OTEL_EXPORTER_OTLP_ENDPOINT when set)
    • Tests in provider_test.go: TestInitProvider_FanOut_GHAWOTLPEndpoints and TestInitProvider_FanOut_TakesPrecedenceOverSingleEndpoint
    • Documented in AGENTS.md under GH_AW_OTLP_ENDPOINTS

    What likely remains

    1. Container configuration — if GH_AW_OTLP_ENDPOINTS isn't being passed to the gateway container in your deployment, that's a deployment/infra change outside the gateway repo itself

    2. JSON-array format — the current implementation expects comma-separated plain URLs. The smoke-otel-tracing CI ([smoke-otel-tracing] Scenario 1 regression: TestInitProvider_NoEndpoint and TestInitProvider_WithHeaders fail when GH_AW_OTLP_ENDPOINTS is set in JSO [Content truncated due to length] #7468) reveals the env var is being set to a JSON-array format ([{"url":"...","headers":"..."}]), which breaks both parsing and existing tests. If per-endpoint headers are needed, resolveExtraEndpoints needs updating to detect and parse JSON input

    3. Per-endpoint headers — currently all endpoints share the same OTEL_EXPORTER_OTLP_HEADERS. The JSON format in [smoke-otel-tracing] Scenario 1 regression: TestInitProvider_NoEndpoint and TestInitProvider_WithHeaders fail when GH_AW_OTLP_ENDPOINTS is set in JSO [Content truncated due to length] #7468 suggests per-endpoint headers may be desired

    Would be helpful to know which of these is the blocking gap! If it's just container config, this issue might already be resolved at the gateway level.

    Warning

    Firewall blocked 1 domain

    The following domain was blocked by the firewall during workflow execution:

    • proxy.golang.org

    To allow these domains, add them to the network.allowed list in your workflow frontmatter:

    network:
      allowed:
        - defaults
        - "proxy.golang.org"

    See Network Configuration for more information.

    Generated by Repo Assist · 1.2K AIC · ⊞ 50.2K · ◷
    Comment /repo-assist to run again

    Add this agentic workflows to your repo

    To install this agentic workflow, run

    gh aw add githubnext/agentics/workflows/repo-assist.md@851905c06e905bf362a9f6cc54f912e3df747d55
    
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions