fix(indexer): wait for the core service before starting the ETL

[rickyrombo]

Follow-up to #1042, which fixed the part of today's stall that turned a 60-second blip into an 8-hour outage. This fixes the blip itself.

The race

The ETL resolves its chain ID from the core service before indexBlocks() starts. Its retry budget is 30 attempts at a flat 2s delay — a fixed ~58 seconds, hardcoded as unexported constants (chainIDInitMaxAttempts, chainIDInitRetryDelay) in go-openaudio/pkg/etl, with InitializeChainID(ctx) exposing no knobs. It can't be raised from this repo without an upstream release.

That budget is shorter than a cold audiusd start. Today's GKE node upgrade recreated every pod at once:

Time (UTC)	Event
10:22:26	creator-1/audiusd-0 created
10:22:33	core-indexer starts polling rpc.audius.co for chain ID
10:23:33	audiusd container actually starts (~67s)
10:23:39	attempt 30 fails → error initializing chain ID after 30 attempts

It missed by about six seconds. Any simultaneous restart is a coin flip.

The fix

Gate etlIndexer.Run() behind a readiness poll on the same core endpoint the ETL will use, with a 15 minute budget. By the time Run() reaches its own retries, the service is up and attempt 1 succeeds.

This is the graceful path, not the safety net. #1042 means a failure here now crashes the process and k8s restarts it — but waiting beats crash-looping through escalating CrashLoopBackOff delays and re-running startup on every pass. If core is still down after 15 minutes, crashing is the right answer, and that is what happens.

Two details worth flagging for review:

• Cancellation is distinguished from timeout. A SIGTERM arriving during the wait returns context.Canceled, which main.go:58 ignores, so shutdown stays quiet. Without that branch the wrapped lastErr would panic on every graceful shutdown during startup — a regression this PR would otherwise have introduced.
• Timeout and poll interval are parameters, not constants read inside the function, mirroring upstream's initializeChainID(ctx, maxAttempts, retryDelay) signature. That's what lets the tests run in milliseconds instead of sleeping for real.

Testing

TestAwaitCoreReady{RetriesUntilCoreIsUp,TimesOut,ReturnsCanceledOnShutdown} against a fake CoreServiceClient that embeds the interface and overrides only GetNodeInfo. The three cases cover the recovery path, the give-up path, and the graceful-shutdown path. All pass in 0.5s; go build ./... and go vet ./indexer/ are clean. The rest of ./indexer/ needs postgres on :21300, which wasn't up locally, so those didn't run.

Still outstanding

core-indexer has no liveness, readiness, or startup probe, and main.go's indexer case doesn't start a health server the way the solana-indexer and eth-indexer cases do. /health_check?max_core_indexer_block_diff=N already returns 500 on excess lag — nothing calls it. That's the remaining reason a wedged indexer goes unnoticed; happy to take it next.

🤖 Generated with Claude Code