[aw-failures] [P0] MCP gateway hard-fails on one unreachable server (Datadog 503) — kills Smoke OTEL #52062

Problem statement

Make the MCP gateway degrade a single unreachable server instead of terminating the whole gateway — one flaky external dependency (Datadog) is taking down Sentry, Grafana, and OTLP export along with it.

Affected workflows and runs

• Smoke OTEL (§31491021479, 2026-08-11T12:23Z) — datadog MCP server ping returned HTTP 503 on all 3 retries (progressive timeouts 10s/20s/30s); gateway logged ERROR: 1 required server(s) failed connectivity check (3 succeeded: grafana, safeoutputs, sentry) and terminated the whole process rather than continuing without datadog.
• Same signature previously on §31023914914 (2026-08-05) and §30606870031 (2026-07-31) — 3 occurrences in the last two weeks.
• Baseline for comparison (success): §31264782373 (2026-08-08).

Probable root cause

The gateway's startup connectivity check treats every configured MCP server as equally "startup-critical." When datadog fails its ping/initialize check after retries, the gateway aborts before the agent job even starts — even though 3 of 4 servers (grafana, safeoutputs, sentry) connected successfully. audit-diff between the failed run and the last success confirms the blast radius: the failed run made zero calls to sentry.io, grafana.net, or the OTLP intake host, all of which were exercised 13-15 times each in the successful baseline. The other 3 integrations never got a chance to run.

Proposed remediation

1. Add a per-server criticality flag (or default to "best-effort") in the gateway's startup check config so a single non-essential server's failure logs a warning and continues, instead of calling process.exit/terminating the gateway.
2. For servers that must stay required, add a circuit-breaker/backoff so a transient 503 doesn't repeat the full 10s/20s/30s retry sequence within the same 6h window.
3. If datadog genuinely needs to be startup-critical for this workflow's test purpose, at minimum surface a distinct, actionable log line (e.g. "non-critical MCP server unavailable, continuing without it") vs. the current blanket "Gateway process will be terminated."

Success criteria / verification

• A single unresponsive non-critical MCP server (e.g. datadog returning 503) no longer aborts gateway startup — the other configured servers (grafana, safeoutputs, sentry) continue to be reachable and used by the agent.
• No new "ERROR: MCP server checks failed - no servers could be connected" gateway-termination failures on Smoke OTEL for at least 2 consecutive scheduled runs after the fix ships.
  Related to [aw-failures] [aw] Failure Investigator Report — 2026-08-11 13:00 (6h) #52061

Generated by 🔍 [aw] Failure Investigator (6h) · agent · 124.7 AIC · ⌖ 42.2 AIC · ⊞ 5.3K · ◷

• expires on Aug 18, 2026, 5:20 AM UTC-08:00