feat(executor): implement DPIPE [CLAUDE] - #8146
Conversation
| if isinstance(this, list) or isinstance(e, list): | ||
| this = this if isinstance(this, list) else [this] | ||
| e = e if isinstance(e, list) else [e] | ||
| return this + e |
There was a problem hiding this comment.
Why are we overloading || to concatenate lists here? What dialect's semantics are you implementing?
There was a problem hiding this comment.
I was looking at a combination of postgres/sqlite and duckdb. I've just pushed a new update which is mostly (but not entirely) consistent with postgres. The different dbs vary here - I'd prefer a behavior closer to postgres, but I think there are a few different choices you could make here (but it does appear you have to trade something away in any choice).
| this = this if isinstance(this, list) else [this] | ||
| e = e if isinstance(e, list) else [e] | ||
| return this + e | ||
| return safe_concat(this, e) |
There was a problem hiding this comment.
Should this be safe_concat only if "safe" is truthy? Otherwise I think we'd need to match the semantics of the source dialect and crash if there's a non-string input.
There was a problem hiding this comment.
Yes, I think test_dpipe_source_dialect_coercion should now cover that?
dbf6b63 to
4e1ae9c
Compare
DPIPE was absent from the executor's ENV, so `||` raised `name 'DPIPE' is
not defined` rather than answering. Every dialect that spells concatenation
that way reaches it -- postgres, sqlite, duckdb, oracle, tsql, bigquery and
the default. MySQL is the exception, where `||` parses to Or rather than
DPipe, so nothing changes there.
For scalars this is exactly SAFECONCAT, which already coerces its operands
the way `||` does ('a' || 1 is 'a1', where CONCAT raises on the int). Rather
than write that twice, SAFECONCAT's lambda becomes a named safe_concat that
both use.
Arrays are the part it adds, and the reason DPIPE cannot simply be routed to
SAFECONCAT in the generator: `||` concatenates them too, and SAFECONCAT
would stringify the operands and answer '[1, 2][3]' -- a plausible wrong
string rather than an error.
'a' || 'b' -> 'ab'
'a' || 1 -> 'a1'
ARRAY[1,2] || ARRAY[3] -> [1,2,3]
ARRAY[1,2] || 3 -> [1,2,3]
'a' || NULL -> NULL
All checked against a real PostgreSQL 18.
4e1ae9c to
6c8444a
Compare
v30.17.0 carries four executor fixes pg_mimic was compensating for: LIKE anchoring and metacharacter escaping (44b73a00), the negate flag on LIKE/ILIKE (#8139), ILIKE itself (#8144) and LENGTH (#8145). Their tripwires XPASSed, which is what the strict xfail is for. Removed from examples/git_sql.py: the ENV["LIKE"]/ENV["ILIKE"] replacements, the LENGTH fill, and _unfold_negations. pushdown() carries its own negate guard, so nothing there depended on the rewrite. The shipped LIKE also applies re.DOTALL, which the replacement did not -- so `%` now spans newlines in a commit body, as Postgres does. DPIPE stays: it is still in review as tobymao/sqlglot#8146. The four tests lose the xfail mark rather than being deleted, matching what test_full_outer_join_preserves_unmatched_rows already does -- they document what the floor buys and catch a regression or a careless downgrade. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
DPIPEis absent from the executor'sENV, so||raises rather than answering:Approach
No
DPIPEfunction is added. The generator routes||to functions that already exist, choosing by the operand's annotated type:The source dialect is not available at codegen — the executor generates from the optimized tree — so
safe(not STRICT_STRING_CONCAT) is the only source-dialect signal that survives parsing. It decides whether a non-string operand coerces or raises. Types come from the optimizer, whichexecute()already runs;execute()also infers them fromtables=.ARRAYCONCAT
exp.ArrayConcatwas already generatingARRAYCONCAT(...)via_renameinto anENVwith no such key, soARRAY_CONCATandARRAY_CATraisedNameErroras well. The one new function fixes those too.Behaviour, and one open choice
arrayconcatwraps a non-list operand so that an element appends to an array. That single line is the one place this diverges by dialect, so both versions are shown. Checked against PostgreSQL 18.4, SQLite 3.51 and DuckDB 1.5.5:'a' || 'b'ababababab'a' || 1a1a1a1a1a1'a' || NULLNULLNULLNULLNULLNULLARRAY[1,2] || ARRAY[3][1,2,3][1,2,3]{1,2,3}[1,2,3]ARRAY[1,2] || 3[1,2,3]{1,2,3}3 || ARRAY[1,2][3,1,2]{3,1,2}1 || 212121212There is no dialect-neutral option: the engines disagree, and the executor has one
ENVshared by all of them withsafeas the only dialect knob the parser records. Dropping the wrap does not remove a dialect-specific behaviour, it selects DuckDB's instead of Postgres'. So the question is which divergence to take.Written as it is,
||mostly follows Postgres.The stricter alternative is available rather than impossible: the same type information would support raising wherever any of the three engines would — both operands known-numeric gives Postgres' rejection of
1 || 2, an array beside a bare element gives DuckDB's. It would be best-effort, since operands areUNKNOWNwhere nothing annotates them, and it has to test for known-numeric rather than for not-text, because a string literal is not always annotated either. I have not taken it because it is more conservative than the surrounding executor, not because it cannot be done:1 || 2lands as'12'here by routing to the existingSAFECONCAT, which is right for theCONCATfunction — Postgres coerces there too,concat(1,2)is12— though Postgres'||operator does require one operand to already betext.MySQL is unaffected throughout —
||is logical OR there, so sqlglot parses it toOr, notDPipe. Under presto/trino — the only dialects settingSTRICT_STRING_CONCAT—'a' || 1raises instead, matching the engine.Full suite: 1321 tests, OK.