fix(ecs): share single AWS client across all ECS plugin executions#119
Open
Will Graham (wlggraham) wants to merge 1 commit into
Open
fix(ecs): share single AWS client across all ECS plugin executions#119Will Graham (wlggraham) wants to merge 1 commit into
Will Graham (wlggraham) wants to merge 1 commit into
Conversation
Previously, each ECS job execution called config.LoadDefaultConfig and constructed new ecs.Client and cloudwatchlogs.Client instances in buildExecutionContext and PullLogs. Under high concurrency (e.g. many parallel amazon_serp_backfill_replay shards), this caused N independent HTTP connection pools and TLS sessions to accumulate in the heimdall process, contributing to OOM kills. Clients are now initialized once in New() on the commandContext and reused across all Execute() calls. The AWS SDK v2 clients are goroutine-safe so no locking is required. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
prasadlohakpure
approved these changes
Jul 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
ecs.Clientandcloudwatchlogs.Clientare now initialized once inNew()oncommandContextand reused across allExecute()calls, rather than being created fresh per jobPullLogsnow accepts a*cloudwatchlogs.Clientparameter instead of constructing its ownbuildExecutionContextassigns the shared clients fromcommandContexttoexecutionContextBackground
Diagnosed via a prod OOM incident on
heimdall-task-definition:422. Two ECS task instances of the Heimdall server were OOM-killed within ~68 minutes of each other. In both cases, multipleamazon_serp_backfill_replay_*shards (largecaterpillar-task-ljobs, 4096 CPU / 8192 MB) were running concurrently.The root cause: each concurrent ECS job called
config.LoadDefaultConfigand created independentecs.Clientandcloudwatchlogs.Clientinstances with their own HTTP connection pools and TLS sessions. With many parallel shards in-flight, these accumulated in the Heimdall process until the kernel OOM-killed it.The AWS SDK v2 clients are goroutine-safe, so sharing a single instance across all executions of the same command is safe and idiomatic.
Test plan