Repository navigation
fix(#11838): describe schemaless columns of a prepared statement from its rows, and bind a uuid parameter - #11920
Conversation
|
Warning Review limit reachedYou've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Next included review available in 16 minutes. View limit details
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. |
|
Tick the box to add this pull request to the merge queue (same as
|
Up to standards ✅🟢 Issues
|
| Metric | Results |
|---|---|
| Complexity | 8 |
NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.
|
Triage: Summary: The PR fixes #11838 in three parts, all in
What I checked:
Minor suggestions (not blocking):
No security concerns and no unrelated changes. |
|
Triage: Reviewed the diff for #11838 (postgresw only). I did not run the build or the tests. What it does
Findings
Minor suggestions, not blockers
The one behaviour change to be aware of is that a pgjdbc application can now get |
…me values they wrap (#11979) Fixes #11955. ### Problem openCypher hands out its temporals as `CypherDate`, `CypherLocalDateTime`, `CypherDateTime`, `CypherLocalTime` and `CypherTime`. That covers function results and a stored DATE / DATETIME read as `n.prop`. Bolt unwrapped them to `java.time` in a private `BoltStructureMapper.unwrapCypherTemporal`. The gRPC encoder (`GrpcTemporalCodec.encode`) and the Postgres wire (`PostgresType.getTypeForValue` / `toText` / `serializeAsBinary`) had no arm for them, so they went out as untagged `string_value` and as `varchar`. SQL sends the same property as a tagged `timestamp_value` and as `date` / `timestamp`. ### Change - **Engine:** a new static `CypherTemporalValue.toJavaTime(Object)` returns the `java.time` value a Cypher date, time or datetime wraps, and any other value unchanged (one `instanceof` check for a value that is not a Cypher temporal). `CypherDuration` stays as it is, since each protocol already has its own encoding for it. - **Bolt:** uses it in place of its private `unwrapCypherTemporal`, which is removed. Same behaviour. - **gRPC:** `GrpcTemporalCodec.encode` unwraps first, so a Cypher temporal gets the kind and `logical_type` its `java.time` value already gets (#9969). The codec is shared with the gRPC client's `ProtoUtils`. - **Postgres:** `getTypeForValue`, `toText` and `serializeAsBinary` unwrap first. A Cypher date is now announced as `date` (1082) and a local datetime as `timestamp` (1114). A zoned datetime, time or local time stays `varchar`, as the SQL `ZonedDateTime` / `OffsetTime` / `LocalTime` does. The elements of a list are unwrapped too (`arrayElementToText`, `serializeArrayToString`, `toJsonElement`), so a Cypher `collect(n.dt)` is written as SQL writes the same list. ### Measured (E2E, `proto_repros.py F1`, container with this branch's engine, server, network, grpcw, postgresw and bolt jars) | column | gRPC before | gRPC after | Postgres before | Postgres after | |---|---|---|---|---| | `n.da` (DATE) | `string_value` "2024-02-29" | `timestamp_value` / `date` (same as SQL) | varchar | date 1082 `2024-02-29` (same as SQL) | | `n.dt` (DATETIME) | `string_value` "2024-02-29T13:14:15.123" | `timestamp_value` / `datetime` (same as SQL) | varchar | timestamp 1114 `2024-02-29 13:14:15.123000` (same as SQL) | | `date()` | `string_value` | `timestamp_value` / `date` | varchar | date | | `localdatetime()` | `string_value` | `timestamp_value` / `datetime` | varchar | timestamp | | `datetime(...Z)` | `string_value` | `string_value` / `zoned_datetime` | varchar | varchar (as SQL's ZonedDateTime) | | `localtime()` | `string_value` "10:00" | `string_value` / `local_time` "10:00:00" | varchar | varchar | | `time()` | `string_value` "10:00+01:00" | `string_value` / `offset_time` "10:00:00+01:00" | varchar | varchar | Bolt is unchanged (`Date`, `DateTime`, `Time`). The server log no longer prints the "has no encoding of its own" warning for these classes. ### Tests - `Issue11955GrpcCypherTemporalIT`: the Cypher columns are equal to the SQL ones for the stored properties, and each function result carries its kind and tag. - `Issue11955GrpcCypherTemporalWireTest`: the gRPC converter encodes each wrapper as the `java.time` value it wraps, alone and inside a list or a map. - `PostgresColumnTypeResolutionIT.CypherTemporalColumn` (pgjdbc and a raw wire client): column types DATE / TIMESTAMP, the decoded `LocalDate` / `LocalDateTime` values, a list of datetimes written as SQL writes it, and the binary result format. - `PostgresTypeTest.CypherTemporalValues` and `CypherTemporalValueToJavaTimeTest`: the type, the text and the binary encoding of every wrapper on the Postgres wire (alone, in a list, in a nested list), and the engine helper. - They fail on main: the gRPC test where `n.da` from Cypher differs from SQL (`string_value`), the Postgres one on `n.da` typed VARCHAR (12) instead of DATE (91), the list one on `{2024-02-29T13:14:15.123}` instead of `{"2024-02-29 13:14:15.123000"}`. They pass with the fix. ### Overlap #11967 (#11958) also edits `PostgresType.java`, but in the json rendering (`toText`'s record/map branch and the json array helpers). This PR adds one line at the top of `toText`, `serializeAsBinary` and `getTypeForValue`, and one in each of the three list element helpers. If git reports a conflict it is textual, and either PR can merge first. Not related to #11920. #11968 (#11960) conflicts with this PR in the `PostgresType.java` import block: #11968 adds the `CypherDuration` import and this PR adds `CypherTemporalValue` on the same line. The conflict is textual only. Whichever PR merges second needs a rebase that keeps both imports. --------- Co-authored-by: lvca <l.garulli@arcadedata.com>
…oes (#11967) Fixes #11958. ### Problem A Postgres-wire json column holding a record (`@this`, a Cypher node) or a map (`properties(n)`, a map literal) was rendered by `record.toJSON(true)` or `new JSONObject(map)`. Temporals came out as epoch numbers: a DATE as epoch days in a record and epoch milliseconds in a map, and a map dropped the zone of a ZONED_DATETIME and the sub-millisecond digits of a DATETIME_NANOS. HTTP sends the same row as formatted text. ### Change - `PostgresType` renders a record, a result row, a map or a path through `JsonSerializer`, the serializer the HTTP API uses, wherever a json column carries one: the scalar column, an element of a json column holding an array (a path included, e.g. `collect(p)`), and an element of a `text[]` or `json[]` array in text and binary format. Vertex edge counts are left out, as `toJSON(true)` left them out. - The date patterns come from the database the value belongs to: the record's, or the row's (`Result.getDatabase()`) when the row is not an element. A value no database owns (a map literal, a row built without one) uses the configured default patterns. - One `JsonSerializer` instance serves every connection (`PostgresType.JSON_DOCUMENT_SERIALIZER`). It is configured once in the static initializer and never changed, and a serialization keeps no state in the serializer (date patterns and type hints live in locals of each call), the same pattern as gRPC's shared `FAST`/`SAFE` serializers. No per-value allocation of a serializer on the row path. - A record that is not a document (an edge segment, a stripe directory, the payload of an EXTERNAL property) keeps its own `toJSON(true)`: `JsonSerializer` has no form for those and would hand the record back, so the column would carry its `toString()`. `Issue11958JsonDocumentShapesTest.aRecordThatIsNotADocumentKeepsItsOwnJson` pins it. - `JsonSerializer` gets a public `serializeValue(Database, Object)` that delegates to the existing private `serializeValue(database, value, null)`. The schema pattern lookups now go through two helpers that fall back to `GlobalConfiguration.DATE_FORMAT` / `DATE_TIME_FORMAT` when the database is null. Every HTTP call site passes a database, so their output is unchanged; `Issue11958SerializeValueTest.httpRowRenderingIsUnchanged` pins the HTTP rendering of these values. ### Behaviour change for clients (release notes) This changes what a Postgres client receives in a json column. Scalar temporal columns (DATE, TIMESTAMP OIDs) are unchanged. - **Temporals inside json documents are text, not epoch numbers.** A client that read a DATE or DATETIME inside a json column as a number (epoch days or epoch milliseconds) must parse the formatted text instead, the same text and the same schema `dateFormat` / `dateTimeFormat` the HTTP API sends. - **Paths** (`MATCH p = ... RETURN p`) keep the `{_type, nodes, relationships, length}` object with the same members in the same order. Each node and relationship gains the record attributes the HTTP API sends with it: `@rid`, `@type`, `@cat`, plus `@in` / `@out` on a relationship. `path.toMap()` wrote each record with `toJSON(false)`, which left them out. Non-temporal property values are the same. Measured in `aPathKeepsTheShapeOfItsMapAndGainsTheRecordAttributes`: - before: `{"_type":"path","nodes":[{"name":"a","n":7,"da":19782,"dt":1709212455123,"dtn":"2024-02-29 13:14:15.123456789",...}],"relationships":[{"da":19782,"w":2}],"length":1}` - after: `{"_type":"path","nodes":[{"@Rid":"#1:0","@type":"T11958","name":"a","n":7,"da":"2024-02-29","dt":"2024-02-29 13:14:15.123","dtn":"2024-02-29 13:14:15.123456789",...,"@cat":"v"}],"relationships":[{"@Rid":"#4:0","@type":"E11958","da":"2024-02-29","w":2,"@cat":"e","@in":"#1:1","@out":"#1:0"}],"length":1}` - **Rows that are not elements** (a projection such as `SELECT da AS d, name FROM T` yields one) keep the members and the order of `result.toJSON()`; only the temporals change. Measured in `aRowThatIsNotAnElementKeepsTheShapeOfItsJson`: - before: `{"name":"a","n":7,"da":1709164800000,"dt":1709212455123,"dtn":1709212455123,"nested":{"da":1709164800000}}` - after: `{"name":"a","n":7,"da":"2024-02-29","dt":"2024-02-29 13:14:15.123","dtn":"2024-02-29 13:14:15.123456789","nested":{"da":"2024-02-29"}}` - **DECIMAL still differs from HTTP.** A DECIMAL in those documents is unchanged: a JSON number with its exact digits (`12345678901234567890.123456789`), where HTTP sends a string. ### Measured (E2E, psycopg 3.3.9, container with this branch's engine and postgresw jars) `SELECT @this AS rec, {'da': da, 'dt': dt, 'dtn': dtn, 'zdt': zdt} AS m FROM T` and `{cypher}MATCH (n:T) RETURN n, properties(n) AS props`: | property | before, record | before, map | after, record and map (same as HTTP) | |---|---|---|---| | DATE | `19782` | `1709164800000` | `"2024-02-29"` | | DATETIME | `1709212455123` | `1709212455123` | `"2024-02-29 13:14:15.123"` | | DATETIME_NANOS | `"2024-02-29 13:14:15.123456789"` | `1709212455123` | `"2024-02-29 13:14:15.123456789"` | | ZONED_DATETIME | `"2024-03-10T03:30-04:00[America/New_York]"` | `1710055800000` | `"2024-03-10T03:30-04:00[America/New_York]"` | This E2E table was measured on the first commit; the follow-up commit's paths are covered by the tests below, not by a second psycopg run. ### Tests - `Issue11958JsonColumnTemporalsIT` (pgjdbc): a record and a map in SQL and Cypher (fails on main: DATE `19782` vs `"2024-02-29"`); a Cypher path (`MATCH p = ... RETURN p`); Cypher lists of nodes, maps and paths (`collect(n)`, `collect(properties(n))`, `collect(p)`); a map inside a `text[]` (`SELECT ['text', {...}]`); the rows of a subquery (`SELECT (SELECT da, dt, dtn, zdt FROM T) AS rows`). - `Issue11958JsonDocumentShapesTest` (postgresw, embedded): a json column holding a list of records, maps, nested lists and paths; a json document inside a `text[]` literal and inside binary `text[]` and `json[]` arrays; path and non-element row shapes compared with `path.toMap()` and `result.toJSON()`; a non-element query row rendered with its database's custom `dateFormat`; the non-document record branch. - `Issue11958SerializeValueTest` (engine): pins the HTTP row rendering and checks `serializeValue` matches it, with and without a database. - Run on the latest commit: `mvn -o -pl postgresw -am verify -DskipITs=false -Dtest='com/arcadedb/postgres/**/*Test,Issue11958SerializeValueTest' -Dit.test='com/arcadedb/postgres/**/*IT'`: postgresw unit 1001 run, ITs 624 run, engine 2 run; 0 failures, 0 errors, 0 skipped. ### Overlap with #11920 #11920 (#11838) also edits `PostgresType.java`, in the Bind parameter decoding (`deserializeText` / `deserializeBinary`, UUID). This PR touches only the json rendering of result values (`toText`, `arrayElementToText`, the json array helpers), so the two do not change the same code. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **Bug Fixes** * Fixed JSON serialization of date and date-time values when records are returned through PostgreSQL, including values nested in maps, lists, arrays, and paths. * Temporal values now retain their expected formatting in database-backed and detached records, including date precision and time zones. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: lvca <l.garulli@arcadedata.com>
…t as the unprepared one does, and bind a uuid parameter
A Describe('P') of a portal of a named statement announced every undeclared property as varchar (text[] for a list), so a client
that prepares a statement (psycopg from its 6th execution, pgjdbc from its 5th) read true as 't' and 42 as '42'. The portal is now
typed from its rows as the unnamed statement and the simple query protocol type it. A layout kept for the statement from those rows
is applied to a later portal executed without a Describe only when its rows have the same types; otherwise the Execute is refused
with 0A000 'cached plan must not change result type' (routine RevalidateCachedQuery), which pgjdbc heals by preparing again.
A uuid parameter (OID 2950, text or binary) is bound as its canonical text.
…tions to PostgresTypeEncodingIT
|
Rebased onto current main (8da9049) to resolve conflicts in PostgresType and the deleted Issue9008And9009MixedTypesIT. Resolution: kept main's new temporal and interval constants alongside this PR's UUID_CODE; main folded the #9008/#9009 test into PostgresTypeEncodingIT, so the #11838 re-describe assertions (column types of u, a missing column a) are ported there. Re-ran Issue11838PreparedSchemalessTypesIT (4), PostgresTypeEncodingIT (23), PostgresDescribeIT (31), PostgresTypeTest (239): green. |
e71f2fc to
f2cb55c
Compare
|
Triage: Thanks for the thorough write-up and tests. The approach is sound. Describing each portal from its own rows fixes the inconsistent Please address
Minor
The changes follow CLAUDE.md on the points I could check: final parameters, no new dependencies, and regression tests present. |
Fixes #11838.
What was wrong
A
Describe('P')of a portal bound from a named statement announced every property the schema does not declare asvarchar(text[]for a list), the "stable layout" from #9009. Drivers name a statement on their own once it has run a few times (psycopg 3 from the 6th execution, pgjdbc from the 5th), so the same query returnedTrue, then't'. Auuidparameter (OID 2950) was refused at Bind:Type with code 2950 not supported for deserializing.The change
resolvePortalColumnsno longer swaps undeclared properties for text when the portal belongs to a named statement, so aDescribe('P')answers the same OIDs as the unnamed statement and the simple query protocol (16, 23, 701, int[]/int8[], 1114). libpq (and so psycopg) sendsBind, Describe('P'), Executeon every execution of a prepared statement, so each execution is described from its own rows.Bind, Executewith no Describe, reusing that layout and asking binary transfer for the numeric columns. The layout is still recorded on the statement (rememberDescribedLayout), now flaggedcolumnsFromRows. A later portal executed under it without its own Describe is materialized (no longer streamed under the layout) and its rows are compared with it: each column must have the type the rows alone would be described with, or be null on every row. Rows that fit are sent under the layout. Rows that do not fit are refused the way PostgreSQL refuses a prepared statement whose result type changed: SQLSTATE0A000, messagecached plan must not change result type, routineRevalidateCachedQuery. pgjdbc reads that routine as an error a new Parse heals (QueryExecutorBase.willHealViaReparse) and runs the statement again, described from its new rows.writeErrorgained an optional routine field for this.Describe('S')is unchanged: it runs before any parameter is bound, so it keeps the schema/sample layout with undeclared properties as text, and that layout is still applied without a check (asyncpg, npgsql and pgx use this path).Trade-off
pgjdbc retries a
RevalidateCachedQueryerror only when the transaction is not already failed: with the defaultautosave=never, inside an explicit transaction block the refusal reaches the application and aborts the block. So a pgjdbc application that, inside one transaction, runs the same prepared statement over rows whose schemaless property changes type between executions now gets this error where it used to get every value as text. That is what PostgreSQL itself does when a prepared statement's result type changes, and declaring the property in the schema avoids it. [INFERENCE from the pgjdbc 42.7.14 source, not run.]Tests
Issue11838PreparedSchemalessTypesIT(new, 4 tests):aNamedStatementDescribesEachPortalAsTheUnnamedOneDoes: the libpq exchange over a raw socket (Parse+Sync, then Bind/Describe('P')/Execute/Sync); for bool, int, double, list and timestamp values the named statement's OID and text equal the unnamed statement's, and the same named statement describes a bool portal as 16 and an int portal as 23.aPortalExecutedUnderAnotherPortalsLayoutIsRefusedWhenItsRowsDoNotFit: the pgjdbc exchange; an undescribed portal whose rows fit the kept layout gets its DataRow, one whose rows do not gets0A000/RevalidateCachedQuery.jdbcKeepsTheTypesOnceItPreparesTheStatement: pgjdbc with default settings, 8 executions per value:getObjectreturns the same class and value every time (Boolean, Integer, Double, int array, timestamp), then values of alternating types on one prepared statement, each read as its own type.aUuidParameterBinds: pgjdbcsetObject(UUID)over 6 executions, and a text uuid declared as OID 2950 in four spellings; an invalid uuid is refused.Failing first: with
postgresw/src/mainfromorigin/main, all 4 fail (3 failures, 1 error:Type with code 2950 not supported for deserializing); with the fix all 4 pass.Issue9008And9009MixedTypesIT.aPreparedStatementKeepsOneLayoutForRowsOfDifferentTypes(pgjdbc defaults, ids 1,1,1,1,1,2,3,1) is updated: it now also checks thatuis described as int4, int8 and float8 for the three records, and it readsaas absent or null for the records that do not have it. Before, every execution from the 6th reused the text layout of record 1, soawas a null column; now the 6th and 7th executions are prepared again and described from records 2 and 3, which have noa, exactly as the unprepared statement describes them. TheprepareThreshold=-1variant (Describe('S')) is unchanged.Module suite (
-pl postgresw, every test undercom/arcadedb/postgres): 1615 run, 0 failures, 1 error, 0 skipped. The error was the 9009 assertion above, before it was updated; after the updateIssue9008And9009MixedTypesITpasses 5/5 andIssue11838PreparedSchemalessTypesIT4/4.End to end (container
arcadedata/arcadedb:26.12.1-SNAPSHOTwith the Postgres plugin, main jarsa7304ab2aa, psycopg 3)The issue's script, main postgresw jar:
With the fix:
Default psycopg auto-prepare, 8 executions of one query over bool/int/double rows, plus a
uuid.UUIDparameter, main:With the fix: