Repository navigation
fix(lvm): share the in-flight capacity check per volume group - #666
Moisés Hidalgo (hidalz) wants to merge 3 commits into
Conversation
When a disk on the node stops answering, vgs gets stuck in uninterruptible sleep and the context timeout cannot kill it. Every GetCapacity retry then started one more stuck vgs, and they piled up in the driver cgroup until it was OOMKilled. Concurrent checks for the same volume group now share one call, and each caller returns on its own deadline.
GetCapacity returns every error as Internal with its text, so a bare "context deadline exceeded" in the logs did not say what the driver was waiting for.
There was a problem hiding this comment.
🟡 Changes recommended
Unbounded waiter retention and cancellation shared across otherwise independent callers remain unresolved.
2 open issues
What changed in this PR
This PR coalesces concurrent capacity checks per volume group to prevent retries from spawning additional vgs processes when a disk stops responding.
Changes:
- Adds a dedicated singleflight group for capacity queries.
- Wraps capacity calculation with shared execution and per-caller cancellation waits.
- Adds a regression test verifying that timed-out retries share one blocked backend call.
Validation was source-only. The changes introduce no persisted state; compatibility with the latest release and exact LVM runtime behavior were not independently verified.
| File | Description |
|---|---|
internal/csi/core/lvm/lvm.go |
Adds per-volume-group capacity coordination state. |
internal/csi/core/lvm/controller.go |
Shares in-flight capacity calculations between callers. |
internal/csi/core/lvm/controller_test.go |
Tests retries against a blocked capacity lookup. |
Review effort: Balanced
| // calls for the same volume group share one lvm call, and each caller returns | ||
| // when that call finishes or when its own ctx is done. | ||
| func (l *LVM) AvailableCapacity(ctx context.Context, vgName string) (int64, error) { | ||
| ch := l.capacityGroup.DoChan(vgName, func() (any, error) { |
| // when that call finishes or when its own ctx is done. | ||
| func (l *LVM) AvailableCapacity(ctx context.Context, vgName string) (int64, error) { | ||
| ch := l.capacityGroup.DoChan(vgName, func() (any, error) { | ||
| return l.availableCapacity(ctx, vgName) |
|
@microsoft-github-policy-service agree company="Algolia" |
| } | ||
| } | ||
|
|
||
| // TODO(sc): use a cache. |
There was a problem hiding this comment.
Old message not part of this PR
There was a problem hiding this comment.
|
Moisés Hidalgo (@hidalz), thanks for reporting this issue and opening the PR. I reproduced similar behavior by stopping disk I/O in a synthetic test ( I think this addresses the GetCapacity case, but other paths that retry or periodically execute LVM commands could still encounter the same issue and lead to an OOM kill. I'm stilling thinking about this issue and thanks a lot for bringing it to our attention |
|
jmclong (@jmclong) Thanks a lot for the feedback, and for your work on it! I'll stay at your disposition in case you need any more reproduction or details on our side then. Best of luck! |


Background
Makes the capacity checks for a volume group share one
vgscall, like it's done byEnsureVolumeGroup.We got a driver OOMKilled on our AKS nodes every few days.
The hypothesis: when a disk on the node stops answering,
vgshangs on it and the timeout can't kill it (because it's in D state), so each GetCapacity retry started one morevgsuntil the driver hit its memory limit.The node load climbed with no CPU use and the driver memory grew with it, until the Azure disk detached. The driver logs show vgs getting killed in the same window.
With this change: a stuck disk now just costs one
vgs. Each caller still returns on its own deadline.Note
the root fix might be the LVM devices file, so that it only reads the driver's disks.
It changes every LVM command, so I left it to you 🙏
Testing:
GetVolumeGroupGOOS=linux go vetpasses