Repository navigation
Deduplicate concurrent pulls for identical model references across pods with node-level singleflight #35
Description
The current pull deduplication key is based on volumeName and mountID, not on model reference.
For static inline volumes, nodePublishVolumeStaticInlineVolume calls worker.PullModel(..., volumeName, "", reference, modelDir, ...), and Worker.pullModel uses:
inflightKey := fmt.Sprintf("pull-%s/%s", volumeName, mountID)
As a result, if two pods on the same node publish inline volumes with the same model reference but different volume names, they will not share the same in-flight pull.
That means on a single node:
two pods requesting the same model reference can still trigger duplicate pulls if they use different volume identities
there should be 2 level pull concurrency control, singleflight by model reference, semaphore for global download parallelism.
- Per-reference deduplication with singleflight
- If multiple requests on the same node pull the same model reference concurrently, only one real pull should run.
- Other requests should wait on the same in-flight result.
- Global concurrency limiting with a semaphore
- If multiple requests pull different model references concurrently, the node should limit how many model pulls can run at once.
- This should prevent uncontrolled fan-out of network and disk IO.
Activity
aftersnow commentedon May 6, 2026
Thanks for raising this. There is already an active PR that addresses the main duplicate-pull problem: #34. It implements digest-based cross-volume pull deduplication so concurrent requests for the same model content on the same node can reuse an existing pull via hardlinks instead of downloading again.
Could you please check whether #34 covers the issue you reported here? There is also an older related issue, #24, which describes the same
volumeName/mountIDinflight-key limitation.xujihui1985 commentedon May 6, 2026
Thanks! I have checked #34, this should work, do we need to take second level concurrent control that is a global semaphore to prevent too many model pulls into consideration?
aftersnow commentedon May 6, 2026
Thanks for checking. I agree that #34 covers the duplicate-pull case for the same model content.
The global semaphore is a separate concern.
pull_config.concurrencycurrently controls concurrency inside a single pull, not the number of concurrent model pulls on a node. I think we should track the node-level pull semaphore as a follow-up item, either in this issue or a separate PR.@rishi-jat, since you were assigned to the older related issue #24, would you be interested in taking this follow-up work?
rishi-jat commentedon May 6, 2026
Yes, I’d be interested in taking the follow-up work for node-level pull concurrency control.
From the discussion, #34 addresses deduplication for identical model content, but it does not limit the number of concurrent model pulls running on a node. A semaphore-based concurrency layer for node-level pull scheduling makes sense to avoid excessive network and disk IO under high parallel workloads.
I’ll investigate the current pull execution flow and propose an implementation approach for node-level pull concurrency limiting as a follow-up to #34.
imeoer commentedon May 7, 2026
I recommend addressing this issue at the layer level rather than the image level.
We can maintain a map like
layer digest -> []pathto enable layer-level cache reuse:layer digest -> []pathinAfterPullLayerhook.layer digest -> []path.This map key / values need also be cleaned up during volume cleanup, when daemonsets are upgraded / restarted, this map can be rebuilt on startup.
For layer-level pull singleinflight, maybe we can keep only one
backend.Newinstance and do singleinflight in modctl. cc @chlinsFor layer-level flow control of network and disk I/O, we can implement global rate limiting using
BeforePullLayer(we can get layer size in the hook func).rishi-jat commentedon May 9, 2026
@imeoer @xujihui1985 - #39 PTAL. thanks!