Skip to content

Native ingest: insert_columns(parallel=True) skips the columnar fast path (the benchmark's LineItem load uses it) #325

Closed

Description

@tae898

Part of #315. Read at the tag 26.11.1 (commit 2f28a328be), nothing run or timed, no speed claim.

Read: on the native wheel, insert_columns(..., parallel=True) skips the columnar fast path. In _native/database.py:1333-1336 the fast path is taken only if not parallel; a parallel call goes through row dicts and insert_many(parallel=True), that is a column-to-row conversion plus JSON text. On the JVM wheel the same call sends the numeric columns across as arrays.

Why it matters: the benchmark's LineItem load (l1_tpc.py:1163) passes parallel=True, so on the native wheel the largest ingest in the paper campaign (122,880 rows per call at tpch1 and tpch10) is the slow row-and-JSON path, while the JVM wheel's is columnar for the numeric columns. The native wheel is the one that should ingest faster, so this is a place where the glue, not the engine, can decide an ingest number.

Proposal: let the columnar entry run with parallel=True (split the columns by rows across the workers instead of falling back), keep the result and the error behaviour, and add a test that spies on the entry point used. Bindings only if the Java side already accepts column chunks from several threads; I did not check that, and if it does not, the Java glue changes and the native library is rebuilt from the unchanged engine commit.

Measure first: a LineItem-shaped table (numeric columns plus four string columns), 122,880 rows per call, warmed, rows per second and CPU per row, native against JVM, before and after, with the interval rule fixed before the run.

Activity

  1. added
    concurrencyThreads, transactions and conflicts
    enhancementNew feature or request
    nativeThe arcadedb package (native library build)
    performanceSpeed or memory of the bindings, with a measurement
    pythonPull requests that update python code
    on Nov 9, 2026
  2. added this to the 26.12.1 milestone on Nov 9, 2026
  3. self-assigned this
    on Nov 9, 2026
  4. added a commit that references this issue on Nov 10, 2026
    cb678b5
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

concurrencyThreads, transactions and conflictsenhancementNew feature or requestnativeThe arcadedb package (native library build)performanceSpeed or memory of the bindings, with a measurementpythonPull requests that update python code

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions