Fix CombinedLock deadlocks & flaky tests

[headtr1ck]

Closes #10787

Description:

Since 2026-09-27, the test-py314 (flaky) CI job hangs until GitHub's 6h job limit. The culprit is TestDask::test_dask_roundtrip: every dask thread ends up waiting on a netCDF/HDF5 SerializableLock, a reader thread in NetCDF4ArrayWrapper._getitem and the writer in NetCDF4ArrayWrapper.__setitem__.

Root cause

CombinedLock removed duplicate locks with tuple(set(locks)), so the order in which it acquired them depended on memory addresses and on insertion order when hash slots collide. The netCDF4 backend builds its reader lock from NETCDFC_LOCK and HDF5_LOCK, and its writer lock from that reader lock plus the file's write lock. With an unlucky address layout, the reader takes NETCDFC → HDF5 while the writer takes HDF5 → NETCDFC. When dask reads the source and writes the target at the same time, each thread holds one lock and waits forever for the other.

NETCDFC_LOCK and HDF5_LOCK are created once at import, so their addresses are fixed for a given environment: it either never happens or deadlocks every time reads and writes overlap. That is why test_dask_roundtrip has long been marked flaky, why it never reproduced locally, and why adding instrumentation made it go away. The only package difference between the last green run and the first hung run was platformdirs 4.11.14 → 4.12.0; importing it shifted the addresses into a bad layout. Nothing in platformdirs itself is wrong.

While debugging I also found a second bug: CombinedLock.acquire(blocking=False) stopped at the first lock it could not take and never released the ones it already held. CachingFileManager.__del__ makes exactly this call when garbage collection frees a manager whose file is still open, so a lock could stay held forever.

Changes

• CombinedLock acquires its locks in one global order, sorted by the identity of the underlying lock, and releases them in reverse.
• Unpickled SerializableLocks wrap the same threading.Lock as the original, so they now count as one lock instead of being acquired twice.
• CombinedLock.acquire releases the locks it already holds when a non-blocking acquire fails.
• Regression tests in test_backends_locks.py. The lock-order test uses fixed hash values to force the reordering; it and the unpickled-lock test fail without the fix.
• CI hardening, so a hung test can't run for 6 hours again:
  • a timeout-minutes: 60 limit on the test matrix (the slowest jobs normally take about 20 minutes)
  • --timeout-method=thread for the flaky job. The default signal method only interrupts the main thread; deadlocked worker threads survive and keep the pytest process from exiting.

Verification

• The flaky job passes on this PR in 1m22s. Every run of it since 2026-09-27 01:46 UTC had hung or crashed on test_dask_roundtrip.

• Built the way the netCDF4 backend builds them, 2 in 20000 reader/writer lock pairs acquired NETCDFC and HDF5 in opposite orders before this change, and 0 after.

• Locally, the lock, file-manager, netCDF4/h5netcdf/dask backend and distributed tests pass, including the flaky-marked ones.

• Tests added

• User visible changes documented in whats-new.rst

AI Disclosure

• This PR contains AI-generated content.
  • I have tested any AI-generated content in my PR.
  • I take responsibility for any AI-generated content in my PR. Tools: Claude Code with Opus 5.5

[This description was updated by Claude Code on behalf of Michael Niklas]

🤖 Generated with Claude Code

[headtr1ck]

We should merge that asap to fix the failing flaky tests.
Not sure if this means we can move this test out of flaky?

[dcherian]

LGTM. I believe there is an xfail around mfdataset and dask that may pass now.

Any thoughts on this assert: #10787?

[headtr1ck]

LGTM. I believe there is an xfail around mfdataset and dask that may pass now.

I will have a look. Thanks.

Any thoughts on this assert: #10787?

I never found this, thanks for spotting. I think @dschwoerer found the root cause of the issue but the assert won't work properly in all cases. E.g. when an unclosed manager is garbage collected, __del__ raises AssertionError but python prints it as "Exception ignored in del", and the file is never closed.

[headtr1ck]

I checked the mfdataset tests, there is one that is entirely skipped. But unfortunately it is not fixed with this PR. I will investigate and open a follow-up PR.
And maybe a second PR to enable strict xfail checking...