Skip to content

Both wheels: a per-row Python validation loop runs before json.dumps in insert_many and create_vertices #318

Closed

Description

@tae898

This comes from a read-only audit of the tag 26.11.1 (commit 2f28a328be); everything below marked read was seen in the code with the file and line, nothing was run or timed, and no speed claim is made. Related closed work that this builds on and does not repeat: #229 (native point reads), #231 (native insert_many ragged lists), #235 (JVM insert_many cyclic GC), #243 (JVM GraphBatch property shapes), #174 (commit sizes).

Read: insert_many and create_vertices walk every row in Python to validate it (type_conversion.py:931-983) and then walk it again in json.dumps. The native wheel also converts columns to rows and back to columns (_native/database.py:1346, _native/bulk.py:57-71), and NativeGraphBatch._create_vertices_json_bulk has no caller (_native/extras.py:232).

Proposal: validate during the single json.dumps pass (a default hook or a typed encoder), and remove the dead code path. Bindings only. Error timing changes subtly (an error appears during encoding, not before the first row is sent), so the existing tests need a look and a test for the first bad row of a large batch.

Measure first: CPU per row of insert_many on a plain scalar table, warmed, before and after. Done when the loop is gone and the measured gain, or the absence of one, is written here.

Activity

  1. added
    nativeThe arcadedb package (native library build)
    performanceSpeed or memory of the bindings, with a measurement
    pythonPull requests that update python code
    on Nov 9, 2026
  2. added this to the 26.12.1 milestone on Nov 9, 2026
  3. self-assigned this
    on Nov 9, 2026
  4. tae898 commented on Nov 12, 2026

    @tae898
    CollaboratorAuthor

    I measured the Python-side step before the JSON text goes to the engine, old against new, as this issue asks: the 26.11.1 tag (2f28a328be) against main (c90b3f9). It is pure Python on one laptop core (Intel Core Ultra X9 388H), no engine, three plain scalar tables (3 columns, 8 columns, 8 columns with a Hangul string), insert_many in chunks of 10,000 rows and create_vertices at 100,000 and 250,000 rows, with numpy and without it (numpy is only an extra). Median CPU time per row over 21, 7 or 5 repeats, in microseconds. A cell only counts as faster or slower when the slowest repeat of one build beats the fastest of the other.

    cell numpy: tag to main no numpy: tag to main
    insert_many, 3 columns, 10,000 rows 0.789 to 0.785 (1.01x, not separated) 0.796 to 0.780 (1.02x, not separated)
    insert_many, 8 columns, 10,000 rows 1.911 to 2.135 (0.90x, slower) 1.903 to 2.181 (0.87x, slower)
    insert_many, 8 columns, Hangul, 10,000 rows 2.082 to 2.264 (0.92x, not separated) 2.069 to 2.230 (0.93x, not separated)
    create_vertices, 3 columns, 100,000 rows 0.774 to 0.698 (1.11x, faster) 0.777 to 0.856 (0.91x, slower)
    create_vertices, 3 columns, 250,000 rows 0.806 to 0.767 (1.05x, not separated) 0.808 to 0.913 (0.88x, slower)
    create_vertices, 8 columns, 100,000 rows 1.897 to 1.953 (0.97x, not separated) 1.823 to 2.350 (0.78x, slower)
    create_vertices, 8 columns, 250,000 rows 1.891 to 2.082 (0.91x, slower) 1.805 to 2.464 (0.73x, slower)
    create_vertices, 8 columns, Hangul, 100,000 rows 1.846 to 1.806 (1.02x, not separated) 1.885 to 2.222 (0.85x, slower)
    create_vertices, 8 columns, Hangul, 250,000 rows 1.945 to 1.981 (0.98x, not separated) 1.962 to 2.443 (0.80x, slower)

    So the gain is absent. With numpy the two builds are within about 11 percent of each other in both directions. Without numpy main is slower in 7 of the 9 cells, clearly, down to 0.73x of the tag's speed. The plain json.dumps of the same rows costs 0.39 to 1.07 microseconds per row, which is about half of the step and cannot go.

    This is only the Python-side step, not end-to-end ingest: the engine, the crossing into it, parsing and the commit are not in it and are usually the larger part. The per-row loop is gone from the hot path, but the step did not get cheaper, so I opened #371 for the slower no-numpy case and am closing this one, since the measurement was what it asked for.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

nativeThe arcadedb package (native library build)performanceSpeed or memory of the bindings, with a measurementpythonPull requests that update python code

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions