fix(agents): heartbeat.global dropped → per-pod firing → codex rate-limit storm - #500
Merged
Merged
Conversation
…-limit storm
normalizeHeartbeat built the moltbot heartbeat object as {every, prompt, target,
session} and SILENTLY DROPPED `global`. So even though the provisioner sets
global:true for preset-with-heartbeat-template agents (reprovision.ts:132),
it never reached moltbot.json — every agent fired a heartbeat PER POD it belongs
to. With ~20 agents across many pods each (× 429 retries), that cascaded into a
sustained codex/LiteLLM rate-limit storm (378 'API rate limit reached' in 25 min
on 2026-06-27) that blocked the whole dev-agent fleet.
Carry `global` (and `fixedPod`) through normalizeHeartbeat in both the K8s and
legacy provisioners. Verified live: setting global:true on the 27 live agents +
restart dropped rate-limit errors from ~15/min to 0.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Incident
The dev-agent fleet was hitting
FailoverError: ⚠️ API rate limit reachedon nearly every heartbeat — 378 occurrences in 25 minutes — blocking all agent work (and the in-session migration validation).Root cause
normalizeHeartbeat(both provisioners) builds the moltbot heartbeat object as{every, prompt, target, session}and silently dropsglobal. The provisioner setsglobal: truefor preset-with-heartbeat-template agents (reprovision.ts:132), but it never reachedmoltbot.json. Result:heartbeat.globalwas unset for all 27 agents → openclaw fired a heartbeat per pod each agent belongs to. ~20 agents × many pods each × 429-retries = the storm. (CLAUDE.md already flagsheartbeat.global: trueas REQUIRED for exactly this reason.)Fix
Carry
global(andfixedPod) throughnormalizeHeartbeatin bothagentProvisionerServiceK8sand the legacyagentProvisionerService.Verified live
Operationally patched
global: trueonto the 27 live agents + restarted the gateway → rate-limit errors dropped from ~15/min to 0. This PR makes that durable so a reprovision can't silently revert to per-pod firing.🤖 Generated with Claude Code