Skip to content

Both wheels: a batched Arrow reader and the Arrow C Stream interface (to_arrow_reader, __arrow_c_stream__) #322

Open

Description

@tae898

Source for the practice: a literature and documentation pass of 2026-11-09 (the Arrow C Data and C Stream interface specifications, the ADBC documentation, the DuckDB, Polars and psycopg documentation, the JPype user guide, and Raasveldt and Muehleisen, VLDB 2017, on result-set transfer); the exact list of 30 sources is in my notes and I will attach it on request. Everything about our code is read at the tag 26.11.1 (commit 2f28a328be); nothing was run or timed, so there is no speed claim. Part of #315.

Read: neither wheel has a way to read a result in batches as Arrow. to_arrow always materialises the whole table, so peak memory grows with the result. The standard in the ecosystem is a reader with a batch size (DuckDB, ADBC and Polars offer one) and an object that implements __arrow_c_stream__, which Arrow consumers (pyarrow, Polars, DuckDB) read without a copy.

Proposal: add to_arrow_reader(batch_size) returning a pyarrow.RecordBatchReader built from the existing batch iteration, and __arrow_c_stream__ on the result so a consumer can take it directly. Python only, no C interface change, for both wheels; the native wheel's rows still come through its current read path until #312 gives it columnar frames.

Measure: peak RSS and seconds for a 10M-row read through to_arrow against the reader, warmed, per-phase memory from the cgroup (not the process high-water mark). Check that DuckDB and Polars read the object directly. Done when the measured peak memory and the check are written here.

Activity

  1. added
    jvmThe arcadedb-jvm package (JPype build)
    nativeThe arcadedb package (native library build)
    performanceSpeed or memory of the bindings, with a measurement
    pythonPull requests that update python code
    on Nov 9, 2026
  2. added this to the 26.12.1 milestone on Nov 9, 2026
  3. self-assigned this
    on Nov 9, 2026
  4. modified the milestones: 26.12.1, 27.1.1 on Nov 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

jvmThe arcadedb-jvm package (JPype build)nativeThe arcadedb package (native library build)performanceSpeed or memory of the bindings, with a measurementpythonPull requests that update python code

Type

No type

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions