ci: move pandas/scipy stubs to pypi deps

[maxrjones]

Description

This PR moves the pandas/scipy stubs to pypi deps so that the test-nightly workflow succeeds, as discussed in #11279. With the previous version, mypy in test-nightly floated to >2 and lacked pip in the env , which caused a lot of failures.

Checklist

• Closes #xxxx
• Tests added
• User visible changes (including notable bug fixes) are documented in whats-new.rst
• New functions/methods are listed in api.rst

AI Disclosure

• This PR contains AI-generated content.
  • I have tested any AI-generated content in my PR.
  • I take responsibility for any AI-generated content in my PR. Tools: {e.g., Claude, Codex, GitHub Copilot, ChatGPT, etc.}

[keewis]

cc @Illviljan

[keewis]

looks good to me, but why does this resolve >100 tests that are failing on main?

[maxrjones]

why does this resolve >100 tests that are failing on main?

Which workflow are you looking at?

[keewis]

the only one that's failing: https://github.com/pydata/xarray/actions/runs/26844047852/job/79159106197?pr=11370 This shows 59 failing tests, compared to 176 in the latest scheduled run on main: https://github.com/pydata/xarray/actions/runs/26791334945/job/78978472251

I think the reason for that is that numpy is installed through conda and not from the scientific-python nightly wheels repo.

[maxrjones]

the only one that's failing: pydata/xarray/actions/runs/26844047852/job/79159106197?pr=11370 This shows 59 failing tests, compared to 176 in the latest scheduled run on main: pydata/xarray/actions/runs/26791334945/job/78978472251

I think the reason for that is that numpy is installed through conda and not from the scientific-python nightly wheels repo.

Good catch, thanks for noticing that. It turns out the stubs require a released numpy which bumps numpy off the nightly wheels:

• scipy-stubs (via optype -> numpy-typing-compat) requires numpy<2.5
• pandas-stubes requires numpy<=2.3.5
• types-networkx pulls a conda numpy

It's not possible to have an upstream environment with nightly numpy and typing across the dependency suite. I have therefore split the upstream test run and upstream type checking:

• test-nightly (the pytest just) stays stub-free and keeps the nightly numpy restoring the failures
• mypy-upstream (a new env for mypy-upstream-dev) carries the typing toolchain + stubs.

The mypy-upstream uses a released numpy, but I don't see a way around that given the pins.

[keewis]

I was thinking that we'd put all the types-* / *-stubs packages in pypi-packages (since that's what mypy would do to install these, anyways), but indeed it appears that numpy-typing-compat has an upper bound through the version of optype (neither pandas-stubs nor types-networkx appear to have an upper bound on numpy, but installing them through conda will introduce an artificial pin on numpy, and potentially the same with hypothesis)

[maxrjones]

I was thinking that we'd put all the types-* / *-stubs packages in pypi-packages (since that's what mypy would do to install these, anyways), but indeed it appears that numpy-typing-compat has an upper bound through the version of optype (neither pandas-stubs nor types-networkx appear to have an upper bound on numpy, but installing them through conda will introduce an artificial pin on numpy, and potentially the same with hypothesis)

yes, it's messy 😞 Are you alright with the current approach in this PR (not having numpy nightly in the mypy checks)?

[keewis]

I'm fine with it personally so I'll go ahead and merge. I've opened #11371 to keep track of the issue.