You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Native ingest: insert_columns(parallel=True) skips the columnar fast path (the benchmark's LineItem load uses it) #325
Part of #315. Read at the tag 26.11.1 (commit 2f28a328be), nothing run or timed, no speed claim.
Read: on the native wheel, insert_columns(..., parallel=True) skips the columnar fast path. In _native/database.py:1333-1336 the fast path is taken only if not parallel; a parallel call goes through row dicts and insert_many(parallel=True), that is a column-to-row conversion plus JSON text. On the JVM wheel the same call sends the numeric columns across as arrays.
Why it matters: the benchmark's LineItem load (l1_tpc.py:1163) passes parallel=True, so on the native wheel the largest ingest in the paper campaign (122,880 rows per call at tpch1 and tpch10) is the slow row-and-JSON path, while the JVM wheel's is columnar for the numeric columns. The native wheel is the one that should ingest faster, so this is a place where the glue, not the engine, can decide an ingest number.
Proposal: let the columnar entry run with parallel=True (split the columns by rows across the workers instead of falling back), keep the result and the error behaviour, and add a test that spies on the entry point used. Bindings only if the Java side already accepts column chunks from several threads; I did not check that, and if it does not, the Java glue changes and the native library is rebuilt from the unchanged engine commit.
Measure first: a LineItem-shaped table (numeric columns plus four string columns), 122,880 rows per call, warmed, rows per second and CPU per row, native against JVM, before and after, with the interval rule fixed before the run.
Activity
Native insert_arrow and insert_dataframe with parallel=True: the same…
Native insert_columns(parallel=True): take the columnar entry (closes A…