[fix](be) Fix wrong result of single-column null-safe eq join on string keys#65975
Open
HappenLee wants to merge 1 commit into
Open
[fix](be) Fix wrong result of single-column null-safe eq join on string keys#65975HappenLee wants to merge 1 commit into
HappenLee wants to merge 1 commit into
Conversation
…ng keys
### What problem does this PR solve?
Problem Summary:
For a single-column null-safe equal join (`<=>`) on a string key, NULL keys produced by expressions may fail to match each other, and the result can even change with the number of BEs or the tablet/bucket distribution.
Minimal reproduction (2 tables, 4 rows, single BE, no aggregation):
create table mini_p (pk int, ch char(10) not null, v tinyint null)
duplicate key(pk) distributed by hash(pk) buckets 1 properties("replication_num"="1");
insert into mini_p values (1,'x',3),(2,'x',NULL);
create table mini_c (pk int, s varchar(100) null)
duplicate key(pk) distributed by hash(pk) buckets 1 properties("replication_num"="1");
insert into mini_c values (1, NULL), (2, 'x3');
SELECT p.pk, c.pk AS cpk FROM mini_p p LEFT JOIN mini_c c
ON CONCAT(COALESCE(p.ch,''), CAST((p.v % 5) AS STRING)) <=> c.s ORDER BY 1, 2;
-- wrong: (1,2),(2,NULL); correct: (1,2),(2,1), because NULL <=> NULL is true
Root cause: for a single-column string join key, the hash join splits the nullable key column into the nested column plus an external null map (`_serialize_null_into_key=false`), and routes null rows to a dedicated null bucket (`init_join_bucket_num`) where they are matched by raw key bytes (`_eq`). The design requires both sides to first normalize the nested data of null rows to the default value via `replace_column_null_data`, but `ColumnString`/`ColumnString64` do not implement it (only `ColumnVector`/`ColumnDecimal` do), so the normalization is a silent no-op. `MethodStringNoCache::init_serialized_keys_impl` then uses the residual bytes left by expression evaluation in the nested column as the key of a null row, so `NULL <=> NULL` matches only when the residual bytes on both sides happen to be equal (e.g. an all-NULL block short-circuits to the default value, while a mixed block leaves evaluated residue). That is why the result depends on block composition, which in turn depends on BE count, tablet distribution and parallelism. Numeric and decimal keys are not affected because their columns implement `replace_column_null_data`; multi-column keys are not affected because the null flag is serialized into the key.
Fix: in `MethodStringNoCache::init_serialized_keys_impl`, when a null map is provided, normalize the stored key of null rows to a canonical empty `StringRef`. Build and probe go through the same function so null keys on both sides are naturally symmetric. Real empty strings are not affected: they are not null, so they are still hashed into normal buckets and never meet the null keys in the dedicated null bucket.
### Release note
Fix wrong results of single-column null-safe equal join (`<=>`) on string keys when the NULL key comes from an expression.
### Check List (For Author)
- Test:
- Regression test: new suite `query_p0/join/test_null_safe_eq_join_string_key` covering minimal reproduction, mixed-block cases compared with the `= OR (both IS NULL)` oracle, and NULL-vs-empty-string collision cases. Passed on a local cluster.
- Unit Test: new case `HashTableMethodTest.testMethodStringNoCacheNullKeyNormalized` (compiled; the local ASAN UT binary crashes at process init due to a pre-existing libomp/openblas issue unrelated to this change, so it could not be executed locally).
- Behavior changed: Yes - `NULL <=> NULL` on single-column string hash join keys now matches correctly; previous results could be wrong or unstable across different BE counts/plan shapes.
- Does this need documentation: No
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Problem Summary:
For a single-column null-safe equal join (
<=>) on a string key, NULL keys produced by expressions may fail to match each other, and the result can even change with the number of BEs or the tablet/bucket distribution.Minimal reproduction (2 tables, 4 rows, single BE, no aggregation):
Root cause: for a single-column string join key, the hash join splits the nullable key column into the nested column plus an external null map (
_serialize_null_into_key=false), and routes null rows to a dedicated null bucket (init_join_bucket_num) where they are matched by raw key bytes (_eq). The design requires both sides to first normalize the nested data of null rows to the default value viareplace_column_null_data, butColumnString/ColumnString64do not implement it (onlyColumnVector/ColumnDecimaldo), so the normalization is a silent no-op.MethodStringNoCache::init_serialized_keys_implthen uses the residual bytes left by expression evaluation in the nested column as the key of a null row, soNULL <=> NULLmatches only when the residual bytes on both sides happen to be equal (e.g. an all-NULL block short-circuits to the default value, while a mixed block leaves evaluated residue). That is why the result depends on block composition, which in turn depends on BE count, tablet distribution and parallelism. Numeric and decimal keys are not affected because their columns implementreplace_column_null_data; multi-column keys are not affected because the null flag is serialized into the key.Fix: in
MethodStringNoCache::init_serialized_keys_impl, when a null map is provided, normalize the stored key of null rows to a canonical emptyStringRef. Build and probe go through the same function so null keys on both sides are naturally symmetric. Real empty strings are not affected: they are not null, so they are still hashed into normal buckets and never meet the null keys in the dedicated null bucket.Release note
Fix wrong results of single-column null-safe equal join (
<=>) on string keys when the NULL key comes from an expression.Check List (For Author)
query_p0/join/test_null_safe_eq_join_string_keycovering minimal reproduction, mixed-block cases compared with the= OR (both IS NULL)oracle, and NULL-vs-empty-string collision cases. Passed on a local cluster.HashTableMethodTest.testMethodStringNoCacheNullKeyNormalized(compiled; the local ASAN UT binary crashes at process init due to a pre-existing libomp/openblas issue unrelated to this change, so it could not be executed locally).NULL <=> NULLon single-column string hash join keys now matches correctly; previous results could be wrong or unstable across different BE counts/plan shapes.