Repository navigation
HADOOP-18639 DockerContainerDeletionTask is not removed from the Nodemanager's statestore when the task is completed - #5425
Closed
Sejin-Hwang wants to merge 1 commit into
Closed
Sejin-Hwang wants to merge 1 commit into
Sejin-Hwang wants to merge 1 commit into
Conversation
|
🎊 +1 overall
This message was automatically generated. |
…manager's statestore when the task is completed.
Sejin-Hwang
force-pushed
the
deletiontask_leak
branch
from
February 22, 2023 16:44
ab08810 to
b9662f3
Compare
|
🎊 +1 overall
This message was automatically generated. |
Contributor
|
We're closing this stale PR because it has been open for 100 days with no activity. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable. |
Willymontaz
added a commit
to criteo-forks/hadoop
that referenced
this pull request
Jul 31, 2026
…from state store on completion (#102) DockerContainerDeletionTask.run() never called deletionTaskFinished(), which is the only thing in the NodeManager that calls NMStateStoreService.removeDeletionTask(). Unlike FileDeletionTask, which finishes its task at the end of run(), every Docker container removal therefore left a permanent row in the LevelDB state store. DeletionService.recover() reloads and re-schedules those rows on every start, from inside serviceInit -- i.e. before the NM accepts new work -- so the backlog grows without bound across the node's lifetime and genuine deletions queue behind it. Observed in production as ~823k DockerContainerDeletionTask instances queued in the DeletionService executor, reappearing in full after every restart, while appcache cleanup starved behind them. Replaying a stale removal is also pure waste: the container is long gone, but LinuxContainerExecutor.removeDockerContainer still forks a privileged container-executor to run docker inspect before deciding it is not removable. Call deletionTaskFinished() from a finally block so the row is removed even if the removal throws, and guard the unchecked LinuxContainerExecutor cast, which would otherwise throw ClassCastException in the executor thread on a node reconfigured to DefaultContainerExecutor with Docker tasks still in the store. No migration is needed: the existing backlog loads one final time, each task executes and removes itself, and the store converges to empty. Reported upstream twice and still unfixed: YARN-11835 (PR apache#8405, open, unreviewed) and HADOOP-18639 (PR apache#5425, auto-closed as stale). Both upstream patches are a bare one-line deletionTaskFinished() outside any try/finally. Co-authored-by: William Montaz <w.montaz@criteo.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of PR
When the DockerContainerDeletionTask is completed, it should be deleted from the NodeManager's statestore, just like the FileDeletionTask.
JIRA : https://issues.apache.org/jira/browse/HADOOP-18639
How was this patch tested?