distributed/tests/test_scheduler.py::test_memory failing on main

I've noticed in many PRs and also on main that distributed/tests/test_scheduler.py::test_memory seems to be failing a lot.

It seems to mostly happen on Windows (all Pythons) ~~and Ubuntu (Python 3.9)~~.

@jacobtomlinson

Thanks for flagging this @jacobtomlinson -- I noticed this started happening a lot over the last couple of days. cc @crusaderky for visibility

Indeed something changed in the workers on Windows.
What the test used to observe before, and still does on Linux and MacOS, is that

immediately after gen_cluster starts a test, there has been exactly 1 heartbeat for each worker (which conveys a reading of the process memory)
the unmanaged memory of the worker doesn't significantly rise afterwards unless you do something to justify it, e.g. submit a large task.

What I'm observing now, on Windows exclusively, is instead that

most times (but not always), there have already been at least 2 heartbeats from each worker by the time the test decorated by gen_cluster starts running
the unmanaged memory on the Nannies keeps rising on its own, by the tune of 100+ MiB, over the next few seconds

It seems to mostly happen on Windows (all Pythons) and Ubuntu (Python 3.9).

I never observed this on Ubuntu. Do you have a link?

@crusaderky

@crusaderky I was thinking of this suite which actually seems to be a different test. I must've misrememebred.

Navigation Menu