[smoke-otel-tracing] Scenario 1 regression: TestInitProvider_NoEndpoint and TestInitProvider_WithHeaders fail when GH_AW_OTLP_ENDPOINTS is set in JSO [Content truncated due to length] #7468

Failing Scenario

Scenario 1: Provider Initialization (internal/tracing/) — 4 tests fail when run in the smoke-otel-tracing workflow.

Failing Tests

Full Test Output

=== RUN   TestInitProvider_NoEndpoint_ReturnsNoopProvider
2026/06/13 06:45:52 internal_logging.go:50: "msg"="otlptrace: parse endpoint url" "error"="parse \"[{\\\"url\\\":\\\"https://o205451.ingest.us.sentry.io/api/4511419678457856/integration/otlp\\\"/v1/traces\": first path segment in URL cannot contain colon" "url"="[{\"url\":\"https://o205451.ingest.us.sentry.io/api/4511419678457856/integration/otlp\"/v1/traces"
2026/06/13 06:45:52 internal_logging.go:50: "msg"="otlptrace: parse endpoint url" "error"="parse \"\\\"headers\\\":\\\"x-sentry-auth=sentry sentry_key=7b9eaff58bd23ee89a2b3c289865a27a\\\"}]/v1/traces\": first path segment in URL cannot contain colon" ...
    provider_test.go:183:
        Error Trace: provider_test.go:183
        Error:       Object expected to be of type noop.TracerProvider, but was *trace.TracerProvider
        Test:        TestInitProvider_NoEndpoint_ReturnsNoopProvider
--- FAIL: TestInitProvider_NoEndpoint_ReturnsNoopProvider (0.00s)

=== RUN   TestInitProvider_WithHeaders/single_well-formed_header
    provider_test.go:636: timed out waiting for OTLP export request — headers test is non-deterministic
--- FAIL: TestInitProvider_WithHeaders/single_well-formed_header (3.04s)

=== RUN   TestInitProvider_WithHeaders/multiple_headers_with_whitespace_trimmed
    provider_test.go:636: timed out waiting for OTLP export request — headers test is non-deterministic
--- FAIL: TestInitProvider_WithHeaders/multiple_headers_with_whitespace_trimmed (3.04s)

=== RUN   TestInitProvider_WithHeaders/malformed_and_empty-key_headers_are_skipped
    provider_test.go:636: timed out waiting for OTLP export request — headers test is non-deterministic
--- FAIL: TestInitProvider_WithHeaders/malformed_and_empty-key_headers_are_skipped (3.04s)

FAIL    github.com/github/gh-aw-mcpg/internal/tracing   9.395s

Root Cause

The CI environment (smoke-otel-tracing workflow) sets GH_AW_OTLP_ENDPOINTS to a JSON array format:

GH_AW_OTLP_ENDPOINTS=[{"url":"https://o205451.ingest.us.sentry.io/api/4511419678457856/integration/otlp","headers":"x-sentry-auth=sentry sentry_key=7b9eaff58bd23ee89a2b3c289865a27a"}]

resolveExtraEndpoints in config_resolver.go splits this string by comma, producing two garbage URL fragments:

1. [{"url":"https://o205451.ingest.us.sentry.io/api/4511419678457856/integration/otlp"
2. "headers":"x-sentry-auth=sentry sentry_key=7b9eaff58bd23ee89a2b3c289865a27a"}]

Both fragments pass resolveEndpoint() (Go's url.Parse is permissive) and get /v1/traces appended. The OTLP exporter construction calls otlptracehttp.New() with these invalid URLs — which succeeds (lazy connection), so len(exporters) > 0. The SDK provider is created and registered globally instead of noop.

Two related issues:

Issue A — Test isolation (TestInitProvider_NoEndpoint_ReturnsNoopProvider)

The test calls tracing.InitProvider(ctx, nil) without first calling t.Setenv("GH_AW_OTLP_ENDPOINTS", ""). The other TestInitProvider_IsEnabled_* tests correctly isolate with t.Setenv. This test assumes the env is unset, which is wrong in the smoke workflow environment.

Issue B — Test isolation (TestInitProvider_WithHeaders)

The table-driven subtests configure an Endpoint pointing to a local httptest.Server, but do not clear GH_AW_OTLP_ENDPOINTS. Since resolveExtraEndpoints takes precedence over cfg.Endpoint when the env var is non-empty, all spans are routed to the Sentry garbage URLs (which return 401), never reaching the local test server. The 3-second timeout fires.

Issue C — resolveExtraEndpoints doesn't parse JSON format

GH_AW_OTLP_ENDPOINTS is being set to a JSON array with per-endpoint url and headers fields (as shown above). The current implementation treats it as plain comma-separated URLs. If the JSON format is intentional for the workflow, resolveExtraEndpoints needs to detect and parse JSON arrays; otherwise the workflow should set the env var in the plain comma-separated URL format.

Environment

• Go version: go1.25.11 linux/amd64
• Workflow run: 27459360247
• GH_AW_OTLP_ENDPOINTS: [{"url":"https://o205451.ingest.us.sentry.io/.../otlp","headers":"x-sentry-auth=..."}]
• OTEL_EXPORTER_OTLP_ENDPOINT: https://o205451.ingest.us.sentry.io/api/4511419678457856/integration/otlp

Other Scenarios

All other scenarios passed:

• ✅ Scenario 2: Parent Context Propagation — 7 tests pass
• ✅ Scenario 3: HTTP Handler Span Creation — 8 tests pass
• ✅ Scenario 4: Tool Call Span Instrumentation — all circuit breaker and callBackendTool tests pass
• ✅ Scenario 5: Header Parsing — TestParseOTLPHeaders and TestResolveHeaders_* all pass
• ✅ Scenario 6: HTTP Backend Auth — all MCP HTTP connection tests pass
• ✅ Scenario 7: Graceful Shutdown — TestInitProvider_WithEndpoint_ReturnsSdkProvider passes
• ✅ Scenario 8: Binary Integration — binary starts with OTel enabled, W3C parent context resolved: traceId=e3dd6ea3d5866246548b640e777a88b9 confirmed in debug output

📡 OTel tracing smoke test by Smoke OTel Tracing

• expires on Jun 20, 2026, 6:50 AM UTC