Repository navigation
Drop PyPy compatibility? #5681
Description
During a call today the issue of dropping PyPy compatibility was brought up.
We currently loosely maintain PyPy compatibility (we don't test with PyPy at all), but some design/process decisions take PyPy into account. For example, currently the cython build of the scheduler requires that all annotations also work in pure python mode so it can run without cythonization. These annotations muddy the code, and don't work as well with type checkers like MyPy. Additionally, avoiding separating out the cython bits of our code makes it harder to understand what bits have been optimized for cython, and also make it harder to perform further c-extension level optimizations.
Since many users are using libraries that don't support PyPy, PyPy is likely mainly useful for speeding up the scheduler (leaving the workers running cpython). I don't know of any users running in this configuration though (most users use the same python environment for both scheduler and workers), and suspect that the maintenance burden outweighs the benefit.
As such, I suggest that we drop our (loose) PyPy compatibility requirement. This would result in no immediate changes to the codebase, but going forward we might make changes that break compatibility.
Activity
jcrist commentedon Jan 21, 2022
Ope, pinging some relevant people from different orgs: @quasiben, @jakirkham, @jrbourbeau, @fjetter, @jsignell (apologies if this issue is relevant to you and I didn't ping you).
jakirkham commentedon Jan 21, 2022
I think @mariusvniekerk has used PyPy support in the past.
Not sure who else. Random issues like this one ( #2734 ) pop-up from time to time suggesting people use this, but have the occasional minor problem.
jakirkham commentedon Jan 21, 2022
cc @mattip (for awareness)
jakirkham commentedon Jan 21, 2022
With Cython annotations, are these only there because of PyPy? At least my understanding originally when working on those was we cared about having pure Python with no Cython as a run option. This can matter outside of PyPy (for example some linters don't work on Cython). Additionally this adds a compilation step for users or packagers. IOW are we now comfortable requiring Cython? Should add this is somewhat separate from PyPy as Cython can also be used with PyPy.
I'm curious if you have some examples of libraries not supported on PyPy that are important. Asking this as conda-forge builds PyPy packages for all sorts of libraries.
Edit: Happy to discuss how the Cython parts can be improved further, but maybe this is best done in another issue.
mattip commentedon Jan 22, 2022
I am missing a bit of background.
fjetter commentedon Jan 24, 2022
We are indeed not publishing any cythonized builds (#4442). We attempted to implement an optional cythonization for the scheduler (
distributed/setup.py
Lines 26 to 30 in 337f152
There are a bunch of issues open about speeding up the scheduler. It's known to be bottlenecked for high task workflows (e.g. once you reach 1M tasks). There's been some extensive profiling ongoing as well. There are actually many tickets in our tracker about this topic. Here are just a few important ones
jakirkham commentedon Jan 25, 2022
Sorry for taking us down a tangent here, I'm doubtful that Cython and PyPy are incompatible. We aren't doing anything that is likely to cause PyPy significant issues (there may be performance differences though).
mrocklin commentedon Mar 30, 2022
What is the status here?
DId we drop pypy support? Were we even maintaining this?
If we wanted to push forward on dropping this, what would be a next step (if there are any)?
jrbourbeau commentedon Mar 30, 2022
We decided to drop support at the monthly community meeting on 2022-02-03
I think the actual code changes for dropping support are fairly minimal. See #6029 and dask/dask#8863
jakirkham commentedon Mar 30, 2022
There may be packaging changes as well. Filed issue ( conda-forge/distributed-feedstock#201 ) to figure that out
jcrist commentedon Mar 31, 2022
Resolved by #6029. Closing.
mrocklin commentedon Mar 31, 2022