Need more guidance on the configurability and scalability for OTel Collector setup #563

Summary

Substrate hardcodes its OTLP endpoint to the GKE Managed OpenTelemetry collector (opentelemetry-collector.gke-managed-otel.svc.cluster.local:4317) in at least nine places across manifests, a CLI flag, and a best-practices doc. That collector is a centrally-deployed Deployment (single gateway tier), the feature is currently in Preview, and it is GKE-only.

We have no documented answer to any of these:

1. Is a single centralized collector Deployment going to scale for Substrate's telemetry shape?
2. Should we instead recommend operators run their own collector, and should it be a DaemonSet?
3. How does an operator point Substrate at their own collector today?
4. What is the recommended collector topology and config?

Today the honest answer to (3) is "patch six manifests and accept that TLS is impossible." That's not a supportable story for anyone running Substrate outside our GKE dev setup.

This issue asks for a written Collector Setup Guide plus the small amount of code/manifest work needed to make that guide actually followable.

Two potential blockers for a bring-your-own collector

1. TLS is impossible. serverboot.go:109,133 hardcode otlptracegrpc.WithInsecure() /otlpmetricgrpc.WithInsecure(), with the comment "GKE managed traces doesn't support validating the TLS certs of the collector." WithInsecure() overrides scheme inference, so setting OTEL_EXPORTER_OTLP_ENDPOINT=https://… will not produce a TLS connection — it will silently stay plaintext. Any operator pointing at an authenticated or off-cluster collector is blocked outright.
2. Protocol is fixed to gRPC. The otlptracegrpc / otlpmetricgrpc exporters are compiled in, so OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf is ignored.