Build and publish macOS wheels via cibuildwheel#294
Conversation
AlexHill
force-pushed
the
macos-wheels
branch
2 times, most recently
from
d54bf1a to
9b18a20
Compare
Add a macos-15 (arm64) runner to the release workflow, with before-all/before-build scripts that port the Linux wheel recipe to macOS. The Boost, casacore and extension compiles all go through ccache, persisted with actions/cache, to save on recompilations between pipeline runs and Python versions. Dependency versions and locations are defined once in cibw_env.sh. The Linux-only CXXFLAGS/LD_LIBRARY_PATH move into [tool.cibuildwheel.linux]. Closes casacore#232.
|
Hi Alex, it's amazing that you're taking a swing at this 🙏🏼👏 I wanted to verify ratt-ru/arcae#211 on the Mac but then I realised that I first need python-casacore on the Mac 🤦🏼♂️😅 Imagine my joy seeing your effort once I started down that rabbit hole again... |
|
Your latest pipeline failure hits the major issue for me with this approach: relying on Homebrew for third-party libraries. It's all coming back to me 😋 Ideally you want the oldest possible In 2024 this got me thinking if a better route won't be leveraging the efforts of the good folks at conda. Conda does care, very much. They target macOS 11.0 on all their libraries (and even tried to stay on 10.9 for as long as possible). There is a casacore feedstock on conda-forge that is well maintained by Peter Williams that we could leverage. The conda Python package format is very similar to a wheel and there is a direct converter (conda-press?), although the symbols use RPATH if I recall and is therefore not delocated. The problem is to break out of the conda ecosystem 😅 Maybe we can use conda instead of Homebrew on the CI machine and then use cibuildwheel on top of that? This is an interesting thread. Or, of course, recompile all third-party libraries (FFTW, gfortran, GSL, WCS, deflate, CFITSIO, HDF5, ...) from source, just using Homebrew for the build tools 😅 |
Isn't that actually a good idea? If using ccache and actions/cache it might be possible to alleviate that cost. From all of those, I suspect gfortran might be the most difficult to tame, but for the rest it shouldn't be a big problem, I think |
cibuildwheel defaults arm64 wheels to 11.0, but the bundled Homebrew libraries are built against the runner's macOS version, which causes build failures at the delocatestep. Derive MACOSX_DEPLOYMENT_TARGET from runner's macOS version.
|
Pinning the target to 15.0 works for me because I'm still on Sequoia (but upgrading any day now) 😁 This does limit the userbase, but given that it's so small, it might be worth the simplicity. Just know that every time GitHub updates their oldest runner, you are potentially forcing end users to upgrade their OS too (or remain stuck on an older casacore and/or Python...). |
|
Hah, on some level I was assuming the pipeline builds on macos-15 would support roughly 26 - 15 versions. I had forgotten that in macOS numbers 26 - 15 = 1!
The source builds do seem like a better long-term plan. That said, some wheels are better than none, and if these stats are remotely accurate, we probably cover a majority of users just with 15 and 26. If you're otherwise happy, perhaps we could figure out the source builds in a subsequent pull request? Thank you for pointing out the pipeline runs in my fork. I was over here hanging out for someone to approve the PR pipeline 🤦 |
|
I like that! I've downloaded your wheels from your pipeline. It works on my macOS 15.7.7 with Python 3.11.3 👏 The versioning is a bit wonky still... I first suspected some shallow or |
|
How would you update the data shipped inside the wheel? By installing a new wheel? 😁 |
Do `import mock_casacore` instead of `import casacore` (or before importing another module that imports casacore) to get a mocked casacore module. This will allow unit tests of the main logic in the mvftoms script without requiring casacore to be installed, which is traditionally bothersome on platforms like macOS and Windows (but see casacore/python-casacore#294 :-) ). I decided to keep `ms_extra.kat_ms_desc_and_dminfo` unmocked and rather mocked the table descriptor calls to casacore inside it, based on the output of the real library. Besides those calls we need to be able to create a Table object with some minimum properties. This qualifies as the first AI-assisted code in katdal. Gemini provided some basic hints on how to mock this, and I rewrote that until it worked.
Do `import mock_casacore` instead of `import casacore` (or before importing another module that imports casacore) to get a mocked casacore module. This will allow unit tests of the main logic in the mvftoms script without requiring casacore to be installed, which is traditionally bothersome on platforms like macOS and Windows (but see casacore/python-casacore#294 :-) ). I decided to keep `ms_extra.kat_ms_desc_and_dminfo` unmocked and rather mocked the table descriptor calls to casacore inside it, based on the output of the real library. Besides those calls we need to be able to create a Table object with some minimum properties. This qualifies as the first AI-assisted code in katdal. Gemini provided some basic hints on how to mock this, and I rewrote that until it worked.
3 participants
Taking another swing at this. @ludwigschwardt @tammojan if you have any time, could you please take a look?
Add macos-15 and macos-15-intel runners to the release workflow, with before-all/before-build scripts that port the Linux wheel recipe to macOS.
The Boost, casacore and extension compiles all go through ccache, persisted with actions/cache, to save on recompilations between pipeline runs and Python versions.
Dependency versions and locations are defined once in cibw_env.sh. The Linux-only CXXFLAGS/LD_LIBRARY_PATH move into [tool.cibuildwheel.linux].
Closes #232, I hope!