Skip to content

Bound leaked ActiveEnumeration collection behind an off-by-default flag + Heartbeat observation telemetry - #2053

Merged
tyrielv merged 1 commit into
microsoft:masterfrom
tyrielv:tyrielv/bound-active-enumeration-leak
Jul 14, 2026
Merged

Bound leaked ActiveEnumeration collection behind an off-by-default flag + Heartbeat observation telemetry#2053
tyrielv merged 1 commit into
microsoft:masterfrom
tyrielv:tyrielv/bound-active-enumeration-leak

Conversation

@tyrielv

@tyrielv tyrielv commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Background

GVFS.Mount crashes surface downstream as STATUS_FILE_SYSTEM_VIRTUALIZATION_UNAVAILABLE (0xC000CE01): once the mount dies, the ProjFS virtualization root is orphaned and every subsequent placeholder access fails. Watson attributes the crashes to radar_high_memory projectedfslib.dll!prjcompletecommand — GVFS.Mount is crashing under memory pressure in the native ProjFS command-completion path.

One suspected managed-side contributor: ProjFS does not always deliver EndDirectoryEnumeration for a StartDirectoryEnumeration (e.g. cancelled enumerations), leaking the ActiveEnumeration — and the projected item list it pins — in activeEnumerations. Over a long-lived mount this grows without bound.

We do not yet have direct evidence that this leak is a dominant driver of the memory pressure. So this PR is deliberately structured to measure first, act second:

What this PR does

1. Observation telemetry — always on.

  • ActiveEnumeration tracks a monotonic LastActivityTickCount.
  • New FileSystemVirtualizer.AddHeartbeatMetadata hook; the Windows virtualizer reports ActiveEnumerationCount / ActiveCommandCount on the periodic Heartbeat event, regardless of the flag below.
  • This ships the signal needed to confirm/refute the leak on real machines.

2. Eviction — off by default, behind gvfs.max-active-enumerations.

  • Unset or <= 0eviction disabled, enumeration hot path unchanged (the default).
  • > 0 → a throttled sweep (≤ once/min, only once the collection exceeds the configured count) evicts enumerations idle longer than 5 minutes.
  • Monotonic clocks for both the throttle and the staleness cutoff; the throttle claim uses Interlocked.CompareExchange keyed on the previously-read tick so two threads in the same interval can't both sweep.

Evicting a live-but-idle enumeration is safe and loud: the next GetDirectoryEnumeration for that id misses the collection and returns HResult.InternalError (fails that one listing rather than returning truncated results as if complete), and every eviction is traced.

Relationship to #2042 (independent — either order)

#2042 addresses a different lever on the same crash: it stops a single failed native ProjFS call from taking down the whole mount (survive-and-report instead of crash). This PR instead targets the memory pressure that triggers those faults, and gates the behavior change off until telemetry justifies it.

The two are independent: this branch does not contain #2042's commit and references none of its code (and vice-versa). They can be reviewed and merged in either order with no rebase — verified with git merge-tree (clean, no conflicts). #2042's own investigation (a local repro that showed enumeration cancellation self-heals and is not the OOM driver) is part of why the eviction here is unproven and therefore gated off by default.

Open questions this is meant to answer

  1. Does activeEnumerations actually grow without bound in the field? (Heartbeat ActiveEnumerationCount.)
  2. If so, does bounding it measurably reduce the native-ProjFS-under-memory-pressure crash rate? (Enable the flag on a ring and compare.)

Testing

  • ActiveEnumeration records activity time.
  • Heartbeat metadata reports the live enumeration count.
  • Eviction disabled → stale enumerations retained.
  • Eviction enabled → stale enumerations evicted (subsequent EndInternalError), freshly-active ones kept.
  • Full unit suite passes (881 passed, 0 failed, 11 skipped).

…t flag

ProjFS does not always deliver EndDirectoryEnumeration for a
StartDirectoryEnumeration (for example when an enumeration is cancelled), which
leaks the corresponding ActiveEnumeration - and the projected item list it pins -
in this.activeEnumerations. Over long-lived GVFS.Mount processes this unbounded
growth is a suspected contributor to the memory pressure behind the native ProjFS
command-completion crashes that surface downstream as
STATUS_FILE_SYSTEM_VIRTUALIZATION_UNAVAILABLE (0xC000CE01).

"Suspected" is the operative word: Watson shows radar_high_memory in
projectedfslib.dll!prjcompletecommand, but we do not yet have direct evidence that
the never-ended-enumeration leak is a dominant driver of that memory. This change
is therefore split into two parts so we can measure before we change the
enumeration hot path anywhere:

1. Observation telemetry (always on). ActiveEnumeration records a monotonic
   LastActivityTickCount (Environment.TickCount64) on creation and on every
   GetDirectoryEnumeration read. WindowsFileSystemVirtualizer now contributes
   ActiveEnumerationCount and ActiveCommandCount to the periodic Heartbeat event
   via a new FileSystemVirtualizer.AddHeartbeatMetadata hook. This ships the
   signal needed to confirm (or refute) that activeEnumerations grows without
   bound on real machines - independent of whether eviction is enabled.

2. Eviction (off by default). When the gvfs.max-active-enumerations git config is
   set to a positive value, a throttled sweep (at most once per minute, and only
   once the collection exceeds the configured count) evicts enumerations idle
   longer than a 5 minute timeout. Config unset or <= 0 disables eviction
   entirely, so the enumeration hot path is unchanged by default. Monotonic clocks
   are used for both the throttle and the staleness cutoff so wall-clock
   adjustments cannot disturb them, and the throttle claim uses an
   Interlocked.CompareExchange keyed on the previously-read tick so two threads
   entering the same interval cannot both sweep.

Evicting a live-but-idle enumeration is safe: the next GetDirectoryEnumeration
for that id misses activeEnumerations and returns HResult.InternalError - it fails
that one directory listing loudly rather than returning truncated results as if
complete - and every eviction is reported via telemetry.

This complements the stabilization-safe "do not crash the mount on a native ProjFS
failure" change (microsoft#2042), which is intentionally kept separate and does not touch
the enumeration hot path. Eviction stays behind the flag until the Heartbeat
telemetry confirms the leak is worth acting on.

Tests: ActiveEnumeration records activity time; Heartbeat metadata reports the
live enumeration count; with eviction disabled stale enumerations are retained;
with eviction enabled stale enumerations are evicted (and their subsequent End
fails with InternalError) while freshly-active ones are kept. Full unit suite
passes.

Assisted-by: Claude Opus 4.8
Signed-off-by: Tyrie Vella <tyrielv@gmail.com>
@tyrielv
tyrielv force-pushed the tyrielv/bound-active-enumeration-leak branch from 0603911 to c8eb49c Compare July 9, 2026 23:01
@tyrielv
tyrielv marked this pull request as ready for review July 9, 2026 23:08
@tyrielv
tyrielv enabled auto-merge July 9, 2026 23:08
@tyrielv
tyrielv merged commit a9a4fe2 into microsoft:master Jul 14, 2026
35 checks passed
@tyrielv tyrielv mentioned this pull request Jul 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants