You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Both wheels: a batched Arrow reader and the Arrow C Stream interface (to_arrow_reader, __arrow_c_stream__) #322
Source for the practice: a literature and documentation pass of 2026-11-09 (the Arrow C Data and C Stream interface specifications, the ADBC documentation, the DuckDB, Polars and psycopg documentation, the JPype user guide, and Raasveldt and Muehleisen, VLDB 2017, on result-set transfer); the exact list of 30 sources is in my notes and I will attach it on request. Everything about our code is read at the tag 26.11.1 (commit 2f28a328be); nothing was run or timed, so there is no speed claim. Part of #315.
Read: neither wheel has a way to read a result in batches as Arrow. to_arrow always materialises the whole table, so peak memory grows with the result. The standard in the ecosystem is a reader with a batch size (DuckDB, ADBC and Polars offer one) and an object that implements __arrow_c_stream__, which Arrow consumers (pyarrow, Polars, DuckDB) read without a copy.
Proposal: add to_arrow_reader(batch_size) returning a pyarrow.RecordBatchReader built from the existing batch iteration, and __arrow_c_stream__ on the result so a consumer can take it directly. Python only, no C interface change, for both wheels; the native wheel's rows still come through its current read path until #312 gives it columnar frames.
Measure: peak RSS and seconds for a 10M-row read through to_arrow against the reader, warmed, per-phase memory from the cgroup (not the process high-water mark). Check that DuckDB and Polars read the object directly. Done when the measured peak memory and the check are written here.
Activity