Skip to content

fix(lvm): share the in-flight capacity check per volume group - #666

Open
Moisés Hidalgo (hidalz) wants to merge 3 commits into
Azure:mainfrom
hidalz:fix/capacity-single-flight
Open

Moisés Hidalgo (hidalz) wants to merge 3 commits into
Azure:mainfrom
hidalz:fix/capacity-single-flight

Conversation

@hidalz

@hidalz Moisés Hidalgo (hidalz) commented Oct 22, 2026 •

Copy link
Copy Markdown

Background
Makes the capacity checks for a volume group share one vgs call, like it's done by EnsureVolumeGroup.

We got a driver OOMKilled on our AKS nodes every few days.

The hypothesis: when a disk on the node stops answering, vgs hangs on it and the timeout can't kill it (because it's in D state), so each GetCapacity retry started one more vgs until the driver hit its memory limit.

The node load climbed with no CPU use and the driver memory grew with it, until the Azure disk detached. The driver logs show vgs getting killed in the same window.

With this change: a stuck disk now just costs one vgs. Each caller still returns on its own deadline.

Note

the root fix might be the LVM devices file, so that it only reads the driver's disks.
It changes every LVM command, so I left it to you 🙏

Testing:

  • Added a unit test with a stuck GetVolumeGroup
  • GOOS=linux go vet passes
  • CI

When a disk on the node stops answering, vgs gets stuck in uninterruptible sleep and the context timeout cannot kill it. Every GetCapacity retry then started one more stuck vgs, and they piled up in the driver cgroup until it was OOMKilled. Concurrent checks for the same volume group now share one call, and each caller returns on its own deadline.
GetCapacity returns every error as Internal with its text, so a bare "context deadline exceeded" in the logs did not say what the driver was waiting for.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unbounded waiter retention and cancellation shared across otherwise independent callers remain unresolved.

2 open issues
What changed in this PR

This PR coalesces concurrent capacity checks per volume group to prevent retries from spawning additional vgs processes when a disk stops responding.

Changes:

  • Adds a dedicated singleflight group for capacity queries.
  • Wraps capacity calculation with shared execution and per-caller cancellation waits.
  • Adds a regression test verifying that timed-out retries share one blocked backend call.

Validation was source-only. The changes introduce no persisted state; compatibility with the latest release and exact LVM runtime behavior were not independently verified.

File Description
internal/​csi/​core/​lvm/​lvm.go Adds per-volume-group capacity coordination state.
internal/​csi/​core/​lvm/​controller.go Shares in-flight capacity calculations between callers.
internal/​csi/​core/​lvm/​controller_test.go Tests retries against a blocked capacity lookup.

Review effort: Balanced

// calls for the same volume group share one lvm call, and each caller returns
// when that call finishes or when its own ctx is done.
func (l *LVM) AvailableCapacity(ctx context.Context, vgName string) (int64, error) {
ch := l.capacityGroup.DoChan(vgName, func() (any, error) {
// when that call finishes or when its own ctx is done.
func (l *LVM) AvailableCapacity(ctx context.Context, vgName string) (int64, error) {
ch := l.capacityGroup.DoChan(vgName, func() (any, error) {
return l.availableCapacity(ctx, vgName)
@hidalz

Copy link
Copy Markdown
Author

@microsoft-github-policy-service agree company="Algolia"

@hidalz
Moisés Hidalgo (hidalz) marked this pull request as ready for review October 22, 2026 14:16
}
}

// TODO(sc): use a cache.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Old message not part of this PR

@hidalz Moisés Hidalgo (hidalz) Oct 27, 2026 •

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi jmclong (@jmclong)

Any news on this? Can we confirm it's an issue in the driver?

Thanks

@jmclong

jmclong (jmclong) commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Moisés Hidalgo (@hidalz), thanks for reporting this issue and opening the PR. I reproduced similar behavior by stopping disk I/O in a synthetic test (dmsetup suspend --noflush --nolockfs), which caused all  vgs commands to hang.

I think this addresses the  GetCapacity  case, but other paths that retry or periodically execute LVM commands could still encounter the same issue and lead to an OOM kill. I'm stilling thinking about this issue and thanks a lot for bringing it to our attention

@hidalz

Copy link
Copy Markdown
Author

jmclong (@jmclong) Thanks a lot for the feedback, and for your work on it! I'll stay at your disposition in case you need any more reproduction or details on our side then.

Best of luck!

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants