Repository navigation
Description
Summary
On the JVM wheel, to_arrow() and to_columns() of a property that holds different kinds of value from row to row return a different type, and in two cases different values, depending on batch_size, that is, on whether the rows fall in one frame or in several. The to_arrow documentation promises "A column's type does not depend on batch_size (#114)", and merge_columns in _columnar.py promises that "the type of a column belongs to the whole result, not to where the frames end". The to_columns lines below are the plainest case: a True that comes back as 1. The native wheel returns the same result at every batch size, so the two wheels also differ.
Documents are schemaless, so a property holding a map with different keys, a list of ints in one row and of floats in the next, or a bool next to a number is ordinary data.
Cause
ColumnBatcher types each frame's column on its own. One frame with mixed values becomes a json column that pa.array(values) types once, with the text of str(v) as the fallback. Several frames are unified in _columnar._unify_arrow_chunk_types and merge_columns, and the two routes disagree:
merge_columnsjoins numpy parts withnp.concatenate, which turns a bool next to an int or a float into1or1.0. In one frame the same column is ajsoncolumn and keeps[1, True]._unify_arrow_chunk_typeshandles int64 with float64 and lists of floats. A map column with different keys (struct<a>next tostruct<b>),list<int64>next tolist<double>, and a bool next to a float all fall through to_cast_all_to_string, which casts with Arrow (true,1) where one frame writesstr(v)(True,1.0). One frame infersstruct<a, b>andlist<double>.- The docstring of
_unify_arrow_chunk_typessays there is no bool/int64 case to widen.
Reproduction
import tempfile
import arcadedb
print(arcadedb.__backend__, arcadedb.__version__)
CASES = {
"int then bool": [1, True],
"float then bool": [1.5, True],
"maps with different keys": [{"a": 1}, {"b": 2}],
"lists of int then float": [[1, 2], [1, 2.5]],
}
with arcadedb.create_database(tempfile.mkdtemp() + "/db") as db:
db.command("sql", "CREATE DOCUMENT TYPE T")
for label, values in CASES.items():
with db.transaction():
db.command("sql", "DELETE FROM T")
with db.transaction():
for n, value in enumerate(values):
doc = db.new_document("T")
doc.set("n", n)
doc.set("v", value)
doc.save()
q = "SELECT v FROM T ORDER BY n"
print("==", label, "| to_list:", [r["v"] for r in db.query("sql", q).to_list()])
for batch_size in (1, 25_000):
cols = db.query("sql", q).to_columns(batch_size=batch_size)["v"]
table = db.query("sql", q).to_arrow(batch_size=batch_size)
print(f" batch_size={batch_size:<6} to_columns={cols.tolist() if hasattr(cols, 'tolist') else cols}"
f" to_arrow={table.schema.field('v').type} {table.column('v').to_pylist()}")Output on arcadedb-jvm 26.12.1.dev0 (built from main at 0c3c7ec, macOS arm64, Python 3.12):
== int then bool | to_list: [1, True]
batch_size=1 to_columns=[1, 1] to_arrow=string ['1', 'true']
batch_size=25000 to_columns=[1, True] to_arrow=string ['1', 'True']
== float then bool | to_list: [1.5, True]
batch_size=1 to_columns=[1.5, 1.0] to_arrow=string ['1.5', 'true']
batch_size=25000 to_columns=[1.5, True] to_arrow=double [1.5, 1.0]
== maps with different keys | to_list: [{'a': 1}, {'b': 2}]
batch_size=1 to_columns=[{'a': 1}, {'b': 2}] to_arrow=string ["{'a': 1}", "{'b': 2}"]
batch_size=25000 to_columns=[{'a': 1}, {'b': 2}] to_arrow=struct<a: int64, b: int64> [{'a': 1, 'b': None}, {'a': None, 'b': 2}]
== lists of int then float | to_list: [[1, 2], [1, 2.5]]
batch_size=1 to_columns=[[1, 2], [1, 2.5]] to_arrow=string ['[1, 2]', '[1.0, 2.5]']
batch_size=25000 to_columns=[[1, 2], [1, 2.5]] to_arrow=list<item: double> [[1.0, 2.0], [1.0, 2.5]]
With the default batch_size of 25,000 the same split happens wherever two kinds meet across a frame boundary, for example a map column whose first 25,000 rows have other keys than the next. On the native wheel (arcadedb 26.11.1.post4) both batch sizes give the right-hand lines. I did not trace why native is consistent. The fix is in the shared _columnar.py because that is where the per-frame types are joined.
Expected
The result does not depend on batch_size: to_columns keeps the Python values to_list returns, and to_arrow gives the type and values that the rows give inside one frame, on both wheels.
Activity
fix(#428): to_arrow and to_columns of differing rows do not depend on…