[Python] Add type checks to Arrow using pyarrow-stubs and initially ignoring failing checks

[rok]

NOTE: this is preliminary prototype work to be used for discussion and not ready to merge to arrow/main.

This adds pyarrow-stubs into arrow monorepo, fixes some type annotation issues and adds a rudimentary CI type annotation test.

[rok]

Things that come to mind after working on this:

• It would be nice to inline as much annotations as possible for maintainability.
• PR currently copies pyarrow-stubs verbatim, but we would probably want to remove docs from pyi files if possible.

[rok]

@paddyroddy @zen-xu

[rok]

(Cross posting from apache#45919)

Things that come to mind after working on this:

• It would be nice to inline as much annotations as possible for maintainability.
• PR currently copies pyarrow-stubs verbatim, but we would probably want to remove docs from pyi files if possible.

[rok]

@kou any thoughts on this?
(it's too early for reviews, just gathering opinions)

[kou]

Thanks for pinging me here.

My random thoughts:

• If we want to merge this one PR, we may need to follow the ASF IP clearance process. See https://incubator.apache.org/ip-clearance/ and search "Apache Arrow" for the past code donation with IP clearance.
• If we want to open a PR per file, we can use the usual development process: Open an issue, open a PR for it, review the PR and merge the PR.

[amoeba]

Thanks for the work on this @rok. A few thoughts that may be retreading previous discussions:

• Can we somehow calculate type coverage? At least for the public interfaces.
• Since inlining annotations is preferred, maybe we should have a section in the PyArrow docs for contributors who might want to help convert stubs to inlined annotations? Is the goal here to inline all type annotations? Should new contributions add stubs or add inline annotations?
• What's the editor support story here? Do any LSPs pick up stubs?
• Should we have a precommit for type checking?

[paddyroddy]

Can we somehow calculate type coverage? At least for the public interfaces.

This seems like it is currently possible with mypy and pyright and a feature request for ty.

[rok]

Thanks for the great questions @amoeba ! I've raised the inlining one in the discussions since we have more eyes there.

• Can we somehow calculate type coverage? At least for the public interfaces.

Seems like a good idea. I'll check @paddyroddy s suggestion.

• What's the editor support story here? Do any LSPs pick up stubs?

My pycharm seems to pick up the stubs from the arrow/python/stubs automatically. PEP-561 specifies distribution mechanisms and I expect editors would follow the spec, but I could be wrong.

• Should we have a precommit for type checking?

That would be nice. We should also include some sort of stub linter, e.g.: flake8-pyi.

[dangotbanned]

That would be nice. We should also include some sort of stub linter, e.g.: flake8-pyi.

If you're familiar with ruff, there's a flake8-pyi group + some others that help you keep up with new typing features/conventions:

• https://docs.astral.sh/ruff/rules/#flake8-pyi-pyi
• https://docs.astral.sh/ruff/rules/#pyupgrade-up
• https://docs.astral.sh/ruff/rules/#flake8-type-checking-tc
• https://docs.astral.sh/ruff/rules/#flake8-future-annotations-fa
• https://docs.astral.sh/ruff/rules/#flake8-annotations-ann

[github-actions]

Thank you for your contribution. Unfortunately, this pull request has been marked as stale because it has had no activity in the past 365 days. Please remove the stale label or comment below, or this PR will be closed in 14 days. Feel free to re-open this if it has been closed in error. If you do not have repository permissions to reopen the PR, please tag a maintainer.