Skip to content

feat(executor): implement DPIPE [CLAUDE] - #8146

Open
jbylund wants to merge 1 commit into
tobymao:mainfrom
jbylund:feat/executor-dpipe
Open

feat(executor): implement DPIPE [CLAUDE]#8146
jbylund wants to merge 1 commit into
tobymao:mainfrom
jbylund:feat/executor-dpipe

Conversation

@jbylund

@jbylund jbylund commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

DPIPE is absent from the executor's ENV, so || raises rather than answering:

>>> execute("SELECT a || b FROM t", tables=t).rows
ExecuteError: Step 'Join: t (...)' failed: name 'DPIPE' is not defined

Approach

No DPIPE function is added. The generator routes || to functions that already exist, choosing by the operand's annotated type:

def _dpipe_sql(self: generator.Generator, e: exp.DPipe) -> str:
    if e.this.is_type(exp.DataType.Type.ARRAY) or e.expression.is_type(exp.DataType.Type.ARRAY):
        return self.func("ARRAYCONCAT", e.this, e.expression)

    return self.func("SAFECONCAT" if e.args.get("safe") else "CONCAT", e.this, e.expression)

The source dialect is not available at codegen — the executor generates from the optimized tree — so safe (not STRICT_STRING_CONCAT) is the only source-dialect signal that survives parsing. It decides whether a non-string operand coerces or raises. Types come from the optimizer, which execute() already runs; execute() also infers them from tables=.

ARRAYCONCAT

exp.ArrayConcat was already generating ARRAYCONCAT(...) via _rename into an ENV with no such key, so ARRAY_CONCAT and ARRAY_CAT raised NameError as well. The one new function fixes those too.

Behaviour, and one open choice

arrayconcat wraps a non-list operand so that an element appends to an array. That single line is the one place this diverges by dialect, so both versions are shown. Checked against PostgreSQL 18.4, SQLite 3.51 and DuckDB 1.5.5:

as written without the wrap postgres sqlite duckdb
'a' || 'b' ab ab ab ab ab
'a' || 1 a1 a1 a1 a1 a1
'a' || NULL NULL NULL NULL NULL NULL
ARRAY[1,2] || ARRAY[3] [1,2,3] [1,2,3] {1,2,3} n/a [1,2,3]
ARRAY[1,2] || 3 [1,2,3] error {1,2,3} n/a error
3 || ARRAY[1,2] [3,1,2] error {3,1,2} n/a error
1 || 2 12 12 error 12 12

There is no dialect-neutral option: the engines disagree, and the executor has one ENV shared by all of them with safe as the only dialect knob the parser records. Dropping the wrap does not remove a dialect-specific behaviour, it selects DuckDB's instead of Postgres'. So the question is which divergence to take.

Written as it is, || mostly follows Postgres.

The stricter alternative is available rather than impossible: the same type information would support raising wherever any of the three engines would — both operands known-numeric gives Postgres' rejection of 1 || 2, an array beside a bare element gives DuckDB's. It would be best-effort, since operands are UNKNOWN where nothing annotates them, and it has to test for known-numeric rather than for not-text, because a string literal is not always annotated either. I have not taken it because it is more conservative than the surrounding executor, not because it cannot be done: 1 || 2 lands as '12' here by routing to the existing SAFECONCAT, which is right for the CONCAT function — Postgres coerces there too, concat(1,2) is 12 — though Postgres' || operator does require one operand to already be text.

MySQL is unaffected throughout — || is logical OR there, so sqlglot parses it to Or, not DPipe. Under presto/trino — the only dialects setting STRICT_STRING_CONCAT'a' || 1 raises instead, matching the engine.

Full suite: 1321 tests, OK.

Comment thread sqlglot/executor/env.py Outdated
Comment on lines +208 to +211
if isinstance(this, list) or isinstance(e, list):
this = this if isinstance(this, list) else [this]
e = e if isinstance(e, list) else [e]
return this + e

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are we overloading || to concatenate lists here? What dialect's semantics are you implementing?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was looking at a combination of postgres/sqlite and duckdb. I've just pushed a new update which is mostly (but not entirely) consistent with postgres. The different dbs vary here - I'd prefer a behavior closer to postgres, but I think there are a few different choices you could make here (but it does appear you have to trade something away in any choice).

Comment thread sqlglot/executor/env.py Outdated
this = this if isinstance(this, list) else [this]
e = e if isinstance(e, list) else [e]
return this + e
return safe_concat(this, e)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be safe_concat only if "safe" is truthy? Otherwise I think we'd need to match the semantics of the source dialect and crash if there's a non-string input.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I think test_dpipe_source_dialect_coercion should now cover that?

@jbylund
jbylund marked this pull request as draft August 12, 2026 20:09
@jbylund
jbylund force-pushed the feat/executor-dpipe branch from dbf6b63 to 4e1ae9c Compare August 12, 2026 20:10
DPIPE was absent from the executor's ENV, so `||` raised `name 'DPIPE' is
not defined` rather than answering. Every dialect that spells concatenation
that way reaches it -- postgres, sqlite, duckdb, oracle, tsql, bigquery and
the default. MySQL is the exception, where `||` parses to Or rather than
DPipe, so nothing changes there.

For scalars this is exactly SAFECONCAT, which already coerces its operands
the way `||` does ('a' || 1 is 'a1', where CONCAT raises on the int). Rather
than write that twice, SAFECONCAT's lambda becomes a named safe_concat that
both use.

Arrays are the part it adds, and the reason DPIPE cannot simply be routed to
SAFECONCAT in the generator: `||` concatenates them too, and SAFECONCAT
would stringify the operands and answer '[1, 2][3]' -- a plausible wrong
string rather than an error.

    'a' || 'b'              -> 'ab'
    'a' || 1                -> 'a1'
    ARRAY[1,2] || ARRAY[3]  -> [1,2,3]
    ARRAY[1,2] || 3         -> [1,2,3]
    'a' || NULL             -> NULL

All checked against a real PostgreSQL 18.
@jbylund
jbylund force-pushed the feat/executor-dpipe branch from 4e1ae9c to 6c8444a Compare August 12, 2026 20:29
@jbylund
jbylund marked this pull request as ready for review August 12, 2026 20:37
jbylund added a commit to jbylund/pg_mimic that referenced this pull request Aug 12, 2026
v30.17.0 carries four executor fixes pg_mimic was compensating for: LIKE
anchoring and metacharacter escaping (44b73a00), the negate flag on LIKE/ILIKE
(#8139), ILIKE itself (#8144) and LENGTH (#8145). Their tripwires XPASSed, which
is what the strict xfail is for.

Removed from examples/git_sql.py: the ENV["LIKE"]/ENV["ILIKE"] replacements, the
LENGTH fill, and _unfold_negations. pushdown() carries its own negate guard, so
nothing there depended on the rewrite. The shipped LIKE also applies re.DOTALL,
which the replacement did not -- so `%` now spans newlines in a commit body, as
Postgres does.

DPIPE stays: it is still in review as tobymao/sqlglot#8146.

The four tests lose the xfail mark rather than being deleted, matching what
test_full_outer_join_preserves_unmatched_rows already does -- they document what
the floor buys and catch a regression or a careless downgrade.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants