Proxy runner triggers StatefulSet rolling update on every restart, causing crash loop

[JAORMX]

Bug

MCPServers with externalAuthConfigRef (embedded auth server) enter a crash loop where the proxy Deployment and backend StatefulSet are continuously rolled.

Symptoms

1. The StatefulSet .metadata.generation increments on every proxy restart (observed going from 2 → 51+)
2. The proxy runner logs show the StatefulSet going from ready_replicas: 1 → ready_replicas: 0 each time it starts
3. After the 5-minute initialize timeout, the proxy reports "workload started successfully" but the transport immediately dies ("transport is no longer running")
4. The proxy enters CrashLoopBackOff

waiting for statefulset to be ready... ready_replicas:1, observed_generation:7, desired_generation:8
waiting for statefulset to be ready... ready_replicas:0
Waiting for MCP server to be ready, endpoint: http://localhost:8080/mcp
initialize not successful, but continuing
workload started successfully, press Ctrl+C to stop
transport is no longer running, attempting automatic restart

MCPServers without externalAuthConfigRef do not exhibit this behavior — their StatefulSets stabilize normally.

To reproduce

1. Create an MCPServer with externalAuthConfigRef pointing to an MCPExternalAuthConfig of type embeddedAuthServer with an OIDC upstream provider
2. Ensure the required ExternalSecrets (ECDSA signing key, HMAC secret) are synced
3. Observe that the proxy Deployment pod enters CrashLoopBackOff
4. Check the StatefulSet generation — it increments on each proxy restart

Environment

• ToolHive operator v0.20.0
• Kubernetes (EKS)

[JAORMX]

Analysis: deploymentNeedsUpdate omits auth server env vars

After tracing through the code, I believe the primary trigger is in mcpserver_controller.go.

deploymentForMCPServer (line ~1133-1146) adds embedded auth server env vars to the proxy Deployment when externalAuthConfigRef is set:

if configName := ctrlutil.EmbeddedAuthServerConfigName(...); configName != "" {
    authServerVolumes, authServerMounts, authServerEnvVars, err := ctrlutil.GenerateAuthServerConfigByName(...)
    env = append(env, authServerEnvVars...)
}

However, deploymentNeedsUpdate (line ~1596-1658) constructs expectedProxyEnv for comparison without including these auth server env vars. The actual container has them, the expected list doesn't — so the function always returns true for MCPServers with externalAuthConfigRef.

This causes the operator to continuously update the Deployment → the proxy pod restarts → it re-applies the StatefulSet → the backend rolls.

This explains why MCPServers without externalAuthConfigRef are unaffected — they have no auth env vars, so the comparison matches and deploymentNeedsUpdate returns false.

[JAORMX]

Analysis: Non-deterministic env var ordering in StatefulSet SSA

A secondary contributing factor is in pkg/container/kubernetes/client.go (lines 352-355). Env vars for the StatefulSet are built by iterating a Go map:

for k, v := range envVars {
    envVarList = append(envVarList, corev1apply.EnvVar().WithName(k).WithValue(v))
}

Go map iteration is non-deterministic, so each proxy restart produces a different env var ordering in the SSA apply configuration. This may cause the API server to update the stored spec, incrementing the StatefulSet generation and triggering a rolling update.

The same pattern exists for container ports (line ~786, iterating ExposedPorts) and service ports (line ~931).

This is latent for all MCPServers but only manifests as a visible problem when the proxy is being continuously restarted (as caused by the deploymentNeedsUpdate issue above).

[JAORMX]

Correction: Deployment is stable, StatefulSet SSA is the sole trigger

After further investigation, the deploymentNeedsUpdate theory was wrong. The proxy Deployment has generation 1 — the operator is not continuously updating it. The proxy restarts are purely from CrashLoopBackOff, not operator-triggered Deployment rollouts.

The loop is:

1. Proxy starts → DeployWorkload() applies StatefulSet via SSA with non-deterministic env var ordering → generation increments → StatefulSet rolls
2. Backend pod dies during rolling update → proxy's container monitor detects it → transport dies → proxy crashes
3. CrashLoopBackOff → proxy restarts → goto 1

All MCPServer StatefulSets have high generation counts (36-88), confirming the non-deterministic ordering issue is universal. Other servers survive because their proxies don't crash when the backend briefly restarts during the roll — the 5-minute init timeout is long enough for the backend to recover. For this particular server, the timing is such that the backend doesn't recover in time, or the transport health check detects the disruption and kills the proxy.

The fix for the non-deterministic ordering would stabilize all MCPServer StatefulSets and eliminate the rolling updates entirely.

[JAORMX]

Analysis: OIDC discovery blocks HTTP server startup

After deeper investigation, the root cause appears to be that the OIDC upstream provider discovery call blocks the HTTP server from starting, preventing the /health endpoint from responding to liveness probes.

The startup sequence in runner.go is:

1. NewEmbeddedAuthServer() (line ~263) — synchronous, blocking
2. Inside that: authserver.New() → newServer() → defaultUpstreamFactory() → NewOIDCProvider()
3. NewOIDCProvider() calls oidc.NewProvider(ctx, config.Issuer) (oidc.go:150) — blocking HTTP call to {issuer}/.well-known/openid-configuration
4. Only after this completes does transportHandler.Start() (line ~459) run, which starts the HTTP server and makes /health available

If the OIDC discovery call is slow (DNS resolution, network latency, provider response time), the HTTP server never starts in time for the liveness probe (initialDelay=30s, period=10s, failureThreshold=3 → killed at ~50s).

This explains why MCPServers with oauth2 upstream type work fine — the OAuth2 path uses explicit authorizationEndpoint and tokenEndpoint from the config, no discovery call needed. The HTTP server starts immediately.

Possible fixes:

• Start the HTTP server (with /health) before the embedded auth server initialization
• Make OIDC discovery async / lazy
• Add a startup probe distinct from the liveness probe to allow more time for initialization