Repository navigation
fix(#420): JVM new_edges sends a subnormal edge property down the per-edge path - #422
Open
shaunpatterson wants to merge 1 commit into
Open
shaunpatterson wants to merge 1 commit into
shaunpatterson wants to merge 1 commit into
Conversation
…wn the per-edge path The untagged branch of GraphBatch.new_edges wrote the rows with plain json.dumps, so a subnormal double reached the engine as text and was stored as a DECIMAL. It now uses json_bulk_dumps, as the tagged branch uses json_bulk_dumps_tagged, and a ValueError starts the per-edge path over.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
GraphBatch.new_edgeswrote the rows of the untagged branch (no date-time, date, Decimal, bytes, list or dict value) with plainjson.dumps. The engine's JSON parser reads the text of a subnormal double such as5e-324as aBigDecimal, so the edge property was stored as a DECIMAL and read back asdecimal.Decimal. The other bulk paths refuse such rows through_dumps_checkedand fall back to the per-value path, which stores the double.The untagged branch now uses
json_bulk_dumps, as the tagged branch usesjson_bulk_dumps_tagged, and the two share one block: aValueErrorstarts the per-edge path over before anything is buffered.Related issues
Closes #420
Performance changes
new_edgeswith one float property on 300,000 edges, three runs each, old and new interleaved: 1.49, 1.41, 1.32 s before and 1.37, 1.37, 1.46 s after. The differences are within the run-to-run spread; the check costs a substring search of the JSON text and runs the per-row walk only on a hit.Docs
None:
how-to/bulk-loads.adocalready saysnew_edgesfalls back for the whole call when a value cannot cross as text.Checklist
test_subnormal_doubles_stay_doubles[...-new_edges], parametrized overinsert_many,create_verticesandnew_edges; the other two entry points pass on the old code)tests/file that usesnew_edgesorgraph_batch, 949 tests on the JVM wheel, thetest_native_*files excluded asconftest.pydoes)uvx pre-commit run --files <changed files>passes