Local Python env Cluster (rebased)

[dometto]

Rebase of #5013

I refer to explanation and discussion in that PR.

#5013 was never finished because merge conflicts appear. That seems a shame: the conflicts were relatively easy to solve. Opening this PR to see if the tests still pass.

I take no credit for @DPeterK's amazing work -- git adds me as a co-author for their commits because of the rebase, I hope that's ok.

[github-actions]

Unit Test Results

See test report for an extended history of previous test failures. This is useful for diagnosing flaky tests.

    40 files  ±  0      40 suites  ±0   14h 51m 30s ⏱️ + 18m 20s
 4 164 tests +  5   3 981 ✅ + 4    178 💤 ±0   5 ❌ + 1 
81 039 runs  +100  76 780 ✅ +82  4 242 💤 +5  17 ❌ +13 

For more details on these failures, see this check.

Results for commit caa9705. ± Comparison against base commit 40fcd99.

♻️ This comment has been updated with latest results.

[dometto]

One failing test remaining, plus some complaints from mypy. This looks fixable, but please let me know if you would still merge this if I got it to work!

[dometto]

These warnings were being thrown by test_job_submission. I'm assuming it's safe to ignore them.

[dometto]

Tests mostly passing:

• Seeing failures on macos and ubuntu, but in the ci1 segment -- the tests for this PR are in not ci1. These are presumbly unrelated, or flukes.
• Seeings failures on windows; it may be that we need to add different ignore rules to pyproject.toml, or that LocalEnvCluster doesn't work on Windows at all.

The remaining linting (mypy) errors are not related to this PR.

[dometto]

This makes me think we cannot support this on Windows, as subprocess support in Windows is only available using the ProactorEventLoop, and distributed appears to need the SelectorEventLoop on Windows.

Would it be ok to raise a NotImplementedError in LocalEnvCluster on Windows, and skip the CI tests?

[dometto]

Looking into subprocess support in Windows I saw that distributed now has a SubprocessCluster, which wasn't there when #5013 was opened. Wondering if LocalEnvWorker and LocalEnvScheduelr should subclass SubprocessWorker and SubprocessScheduler.

EDIT: this would require a refactor of Subprocess, so that subclasses can override the command (to insert the desired interpreter, and the set_env command prefix). At the moment, SubprocessScheduler also assumes that there is a scheduler_file, which LocalEnv does not. So perhaps inheritance here is not worth it.