Skip to content

JVM to_arrow/to_columns: the type of a column of differing rows depends on batch_size #428

Closed

Description

@shaunpatterson

Summary

On the JVM wheel, to_arrow() and to_columns() of a property that holds different kinds of value from row to row return a different type, and in two cases different values, depending on batch_size, that is, on whether the rows fall in one frame or in several. The to_arrow documentation promises "A column's type does not depend on batch_size (#114)", and merge_columns in _columnar.py promises that "the type of a column belongs to the whole result, not to where the frames end". The to_columns lines below are the plainest case: a True that comes back as 1. The native wheel returns the same result at every batch size, so the two wheels also differ.

Documents are schemaless, so a property holding a map with different keys, a list of ints in one row and of floats in the next, or a bool next to a number is ordinary data.

Cause

ColumnBatcher types each frame's column on its own. One frame with mixed values becomes a json column that pa.array(values) types once, with the text of str(v) as the fallback. Several frames are unified in _columnar._unify_arrow_chunk_types and merge_columns, and the two routes disagree:

  • merge_columns joins numpy parts with np.concatenate, which turns a bool next to an int or a float into 1 or 1.0. In one frame the same column is a json column and keeps [1, True].
  • _unify_arrow_chunk_types handles int64 with float64 and lists of floats. A map column with different keys (struct<a> next to struct<b>), list<int64> next to list<double>, and a bool next to a float all fall through to _cast_all_to_string, which casts with Arrow (true, 1) where one frame writes str(v) (True, 1.0). One frame infers struct<a, b> and list<double>.
  • The docstring of _unify_arrow_chunk_types says there is no bool/int64 case to widen.

Reproduction

import tempfile

import arcadedb

print(arcadedb.__backend__, arcadedb.__version__)
CASES = {
    "int then bool": [1, True],
    "float then bool": [1.5, True],
    "maps with different keys": [{"a": 1}, {"b": 2}],
    "lists of int then float": [[1, 2], [1, 2.5]],
}
with arcadedb.create_database(tempfile.mkdtemp() + "/db") as db:
    db.command("sql", "CREATE DOCUMENT TYPE T")
    for label, values in CASES.items():
        with db.transaction():
            db.command("sql", "DELETE FROM T")
        with db.transaction():
            for n, value in enumerate(values):
                doc = db.new_document("T")
                doc.set("n", n)
                doc.set("v", value)
                doc.save()
        q = "SELECT v FROM T ORDER BY n"
        print("==", label, "| to_list:", [r["v"] for r in db.query("sql", q).to_list()])
        for batch_size in (1, 25_000):
            cols = db.query("sql", q).to_columns(batch_size=batch_size)["v"]
            table = db.query("sql", q).to_arrow(batch_size=batch_size)
            print(f"  batch_size={batch_size:<6} to_columns={cols.tolist() if hasattr(cols, 'tolist') else cols}"
                  f"  to_arrow={table.schema.field('v').type} {table.column('v').to_pylist()}")

Output on arcadedb-jvm 26.12.1.dev0 (built from main at 0c3c7ec, macOS arm64, Python 3.12):

== int then bool | to_list: [1, True]
  batch_size=1      to_columns=[1, 1]  to_arrow=string ['1', 'true']
  batch_size=25000  to_columns=[1, True]  to_arrow=string ['1', 'True']
== float then bool | to_list: [1.5, True]
  batch_size=1      to_columns=[1.5, 1.0]  to_arrow=string ['1.5', 'true']
  batch_size=25000  to_columns=[1.5, True]  to_arrow=double [1.5, 1.0]
== maps with different keys | to_list: [{'a': 1}, {'b': 2}]
  batch_size=1      to_columns=[{'a': 1}, {'b': 2}]  to_arrow=string ["{'a': 1}", "{'b': 2}"]
  batch_size=25000  to_columns=[{'a': 1}, {'b': 2}]  to_arrow=struct<a: int64, b: int64> [{'a': 1, 'b': None}, {'a': None, 'b': 2}]
== lists of int then float | to_list: [[1, 2], [1, 2.5]]
  batch_size=1      to_columns=[[1, 2], [1, 2.5]]  to_arrow=string ['[1, 2]', '[1.0, 2.5]']
  batch_size=25000  to_columns=[[1, 2], [1, 2.5]]  to_arrow=list<item: double> [[1.0, 2.0], [1.0, 2.5]]

With the default batch_size of 25,000 the same split happens wherever two kinds meet across a frame boundary, for example a map column whose first 25,000 rows have other keys than the next. On the native wheel (arcadedb 26.11.1.post4) both batch sizes give the right-hand lines. I did not trace why native is consistent. The fix is in the shared _columnar.py because that is where the per-frame types are joined.

Expected

The result does not depend on batch_size: to_columns keeps the Python values to_list returns, and to_arrow gives the type and values that the rows give inside one frame, on both wheels.

Activity

  1. added
    jvmThe arcadedb-jvm package (JPype build)
    pythonPull requests that update python code
    on Nov 21, 2026
  2. added this to the 26.12.1 milestone on Nov 21, 2026
  3. added a commit that references this issue on Nov 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

in progressThe team is actively working on this issuejvmThe arcadedb-jvm package (JPype build)pythonPull requests that update python code

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions