[go-fan] Go Module Review: go.opentelemetry.io/otel (OpenTelemetry Suite) #7972

Module Overview

OpenTelemetry Go — distributed tracing/metrics/logging framework. This project uses 4 OTel packages:

• go.opentelemetry.io/otel v1.44.0 — Core API
• go.opentelemetry.io/otel/sdk v1.44.0 — SDK (TracerProvider, Sampler, BatchSpanProcessor)
• go.opentelemetry.io/otel/trace v1.44.0 — Span/Tracer interfaces
• go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.44.0 — OTLP/HTTP exporter

Current Usage in gh-aw

• Files: 8 production source files in internal/tracing/, plus proxy/handler.go, server/middleware.go, server/unified.go
• Key APIs: sdktrace.NewTracerProvider, noop.NewTracerProvider, otlptracehttp.New, W3C propagation, resource.New with container/PID/host detectors, tracetest.NewInMemoryExporter in tests

Active Spans

Custom attributes: gen_ai.tool.name, gen_ai.conversation.id, gen_ai.agent.id, gen_ai.system (raw key — removed from semconv/v1.41.0), mcp.method, mcp.response.status, rate_limit.hit, gateway.tag

Improvement Opportunities

🏃 Quick Wins

1. Set gen_ai.operation.name on tool call spans — GenAIOperationName is defined in genai_attrs.go but never used. OTel GenAI semconv recommends it on all GenAI spans:

// span_helpers.go: StartToolCallSpan
GenAIOperationName.String("execute_tool"),  // add this

This is the primary grouping key in Honeycomb/Grafana Tempo.

2. Set GenAIAgentName on tool call spans — Also defined but never set. gen_ai.agent.name = "mcp-gateway" completes the GenAI semconv attributes for better search ergonomics.

✨ Feature Opportunities

3. Add span events for pipeline milestones — span.AddEvent() is not used anywhere (confirmed by codebase search). Span events provide timestamped structured entries within a span's timeline:

// After DIFC pre-phase
toolSpan.AddEvent("difc.pre_phases_complete")
// When rate limit detected
toolSpan.AddEvent("rate_limit.detected", oteltrace.WithAttributes(
    attribute.String("reset_at", resetAt.Format(time.RFC3339)),
))
// On access denied (complement to RecordSpanError)
toolSpan.AddEvent("difc.access_denied", oteltrace.WithAttributes(
    attribute.String("reason", denied.EvalResult.Reason),
))

4. Add OTel metrics (RED dashboard) — go.opentelemetry.io/otel/metric is already in go.sum (transitive dep) but unused. Key metrics to add:

• mcp.tool_calls — counter by server_id, tool_name, result
• mcp.tool_call.duration_ms — histogram for P50/P95/P99
• mcp.rate_limit_events — counter by server_id
• mcp.circuit_breaker.state — gauge (0=closed, 1=open)

5. Expose BatchSpanProcessor tuning in TracingConfig — sdktrace.WithBatcher(exporter) uses OTel defaults (5s timeout, 512 max batch). Add batch_timeout_ms and max_batch_size config fields for gateway workloads with burst parallelism.

📐 Best Practice Alignment

6. Guard expensive attribute computation with span.IsRecording() — For complex attribute builds (JSON, string formatting), check span.IsRecording() first. With the noop provider active this returns false and the computation is skipped.

7. Track gen_ai.system semconv evolution — The current raw key attribute.Key("gen_ai.system") with value "mcp" is a pragmatic workaround. Monitor the OTel GenAI registry for re-introduction/standardization.

8. Future-proof Provider struct for metrics — Add a MeterProvider field alongside TracerProvider so callers can access metrics the same way they access the tracer, without API changes later.

Module Summary

✅ TracerProvider with BatchSpanProcessor · ✅ Noop provider (zero-overhead) · ✅ W3C propagation · ✅ Fan-out exporter · ✅ Configurable sampling · ✅ Resource detectors · ✅ GenAI semconv
❌ span.AddEvent() not used · ❌ Metrics SDK unused · ❌ gen_ai.operation.name not set · ❌ gen_ai.agent.name not set

References

• Docs: (opentelemetry.io/redacted)
• GenAI Semconv: (opentelemetry.io/redacted)
• Release notes: https://github.com/open-telemetry/opentelemetry-go/releases/tag/v1.44.0

Recommendations (priority order)

1. [Easy] Add GenAIOperationName.String("execute_tool") to StartToolCallSpan
2. [Easy] Add GenAIAgentName to mcp.tool_call spans
3. [Medium] Add span.AddEvent() at DIFC pipeline boundaries and rate-limit detection
4. [Medium] Add DIFC denial span events with structured attributes
5. [Large] Add OTel metrics for mcp.tool_calls, mcp.tool_call.duration_ms, circuit breaker state
6. [Small] Expose BatchTimeoutMs/MaxBatchSize in TracingConfig

Generated by Go Fan 🐹 | Run §28012229096

Generated by Go Fan · 704 AIC · ⊞ 34.5K · ◷

• expires on Jun 30, 2026, 8:25 AM UTC