You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Support OTLP multi-endpoint fan-out (GH_AW_OTLP_ENDPOINTS) #7444
Reported in github/gh-aw#38901: The MCP gateway exports OpenTelemetry spans only to the single primary OTLP endpoint (OTEL_EXPORTER_OTLP_ENDPOINT). When gh-aw configures multiple OTLP backends via GH_AW_OTLP_ENDPOINTS, non-primary backends receive the gh-aw lifecycle spans but none of the gateway spans (gateway.request, mcp.tool_call, gateway.backend.execute).
Current Behavior
Gateway reads OTEL_EXPORTER_OTLP_ENDPOINT (single endpoint) and initializes one OTLP exporter
GH_AW_OTLP_ENDPOINTS is not passed into the gateway container
Non-primary backends get incomplete traces (0 gateway spans)
Desired Behavior
The gateway should support multi-endpoint OTLP export so all configured backends receive complete traces including gateway/tool-call spans.
Proposed Approach
Accept GH_AW_OTLP_ENDPOINTS (comma-separated list of endpoint URLs) as a new env var
When set, create one OTLP exporter per endpoint and fan out span exports to all
Per-endpoint failure should not block export to other endpoints (partial-failure tolerance)
Fall back to existing single-endpoint behavior when GH_AW_OTLP_ENDPOINTS is not set
Each endpoint may have its own headers (need a format for per-endpoint headers, or use a shared OTEL_EXPORTER_OTLP_HEADERS for all)
Implementation Notes
The tracing setup lives in internal/tracing/ (provider initialization, OTLP HTTP exporter)
Current single-exporter setup: internal/tracing/provider.go
The --otlp-endpoint flag and OTEL_EXPORTER_OTLP_ENDPOINT env var remain the primary/fallback
Fan-out exporter could wrap multiple otlptracehttp exporters behind a composite SpanExporter interface
The comma-separated multi-endpoint fan-out (GH_AW_OTLP_ENDPOINTS) is already implemented in the codebase. Here's a summary of what's there and what remains:
Already implemented
resolveExtraEndpoints() in internal/tracing/config_resolver.go reads GH_AW_OTLP_ENDPOINTS and splits by comma
newFanoutExporter() in internal/tracing/fanout.go wraps multiple sdktrace.SpanExporters with partial-failure tolerance
InitProvider uses GH_AW_OTLP_ENDPOINTS (takes precedence over OTEL_EXPORTER_OTLP_ENDPOINT when set)
Tests in provider_test.go: TestInitProvider_FanOut_GHAWOTLPEndpoints and TestInitProvider_FanOut_TakesPrecedenceOverSingleEndpoint
Documented in AGENTS.md under GH_AW_OTLP_ENDPOINTS
What likely remains
Container configuration — if GH_AW_OTLP_ENDPOINTS isn't being passed to the gateway container in your deployment, that's a deployment/infra change outside the gateway repo itself
Activity
github-actions commentedon Jun 13, 2026
🤖 This is an automated response from Repo Assist.
The comma-separated multi-endpoint fan-out (
GH_AW_OTLP_ENDPOINTS) is already implemented in the codebase. Here's a summary of what's there and what remains:Already implemented
resolveExtraEndpoints()ininternal/tracing/config_resolver.goreadsGH_AW_OTLP_ENDPOINTSand splits by commanewFanoutExporter()ininternal/tracing/fanout.gowraps multiplesdktrace.SpanExporters with partial-failure toleranceInitProviderusesGH_AW_OTLP_ENDPOINTS(takes precedence overOTEL_EXPORTER_OTLP_ENDPOINTwhen set)provider_test.go:TestInitProvider_FanOut_GHAWOTLPEndpointsandTestInitProvider_FanOut_TakesPrecedenceOverSingleEndpointAGENTS.mdunderGH_AW_OTLP_ENDPOINTSWhat likely remains
Container configuration — if
GH_AW_OTLP_ENDPOINTSisn't being passed to the gateway container in your deployment, that's a deployment/infra change outside the gateway repo itselfJSON-array format — the current implementation expects comma-separated plain URLs. The smoke-otel-tracing CI ([smoke-otel-tracing] Scenario 1 regression: TestInitProvider_NoEndpoint and TestInitProvider_WithHeaders fail when GH_AW_OTLP_ENDPOINTS is set in JSO [Content truncated due to length] #7468) reveals the env var is being set to a JSON-array format (
[{"url":"...","headers":"..."}]), which breaks both parsing and existing tests. If per-endpoint headers are needed,resolveExtraEndpointsneeds updating to detect and parse JSON inputPer-endpoint headers — currently all endpoints share the same
OTEL_EXPORTER_OTLP_HEADERS. The JSON format in [smoke-otel-tracing] Scenario 1 regression: TestInitProvider_NoEndpoint and TestInitProvider_WithHeaders fail when GH_AW_OTLP_ENDPOINTS is set in JSO [Content truncated due to length] #7468 suggests per-endpoint headers may be desiredWould be helpful to know which of these is the blocking gap! If it's just container config, this issue might already be resolved at the gateway level.
Warning
Firewall blocked 1 domain
The following domain was blocked by the firewall during workflow execution:
proxy.golang.orgSee Network Configuration for more information.
Add this agentic workflows to your repo
To install this agentic workflow, run