diff --git a/be/benchmark/benchmark_main.cpp b/be/benchmark/benchmark_main.cpp index 582db25ff6d667..7c64fa2729cefb 100644 --- a/be/benchmark/benchmark_main.cpp +++ b/be/benchmark/benchmark_main.cpp @@ -45,6 +45,7 @@ #include "core/data_type/data_type.h" #include "core/data_type/data_type_string.h" #include "parquet/benchmark_parquet_decoder.hpp" +#include "parquet/benchmark_parquet_kernels.hpp" #include "parquet/benchmark_parquet_reader.hpp" #include "runtime/exec_env.h" #include "runtime/memory/mem_tracker_limiter.h" diff --git a/be/benchmark/parquet/AGENTS.md b/be/benchmark/parquet/AGENTS.md index a8667fb43cdf5f..ba52658a9664c3 100644 --- a/be/benchmark/parquet/AGENTS.md +++ b/be/benchmark/parquet/AGENTS.md @@ -6,9 +6,10 @@ benchmark system described in the design document. ## What exists today -The benchmark binary registers two groups: +The benchmark binary registers three groups: - `ParquetDecoder`: native page decoder benchmarks using in-memory encoded pages. +- `ParquetKernel`: isolated SIMD-sensitive decode and predicate kernels. - `ParquetReader`: local-file benchmarks that call the format V2 Parquet reader directly. The relevant files are: @@ -36,13 +37,16 @@ List all Parquet cases and verify the expected registration counts: ```shell be/output/lib/benchmark_test --benchmark_list_tests \ - | grep -E '^Parquet(Decoder|Reader)/' + | grep -E '^Parquet(Decoder|Kernel|Reader)/' be/output/lib/benchmark_test --benchmark_list_tests \ - | grep -c '^ParquetDecoder/' # currently 152 + | grep -c '^ParquetDecoder/' # currently 228 be/output/lib/benchmark_test --benchmark_list_tests \ - | grep -c '^ParquetReader/' # currently 137 + | grep -c '^ParquetKernel/' # currently 80 + +be/output/lib/benchmark_test --benchmark_list_tests \ + | grep -c '^ParquetReader/' # currently 152 ``` When running the binary directly from `be/build_RELEASE/bin`, make sure the JVM and third-party @@ -61,6 +65,12 @@ be/output/lib/benchmark_test \ --benchmark_out=parquet-decoder-smoke.json \ --benchmark_out_format=json +be/output/lib/benchmark_test \ + --benchmark_filter='^ParquetKernel/' \ + --benchmark_min_time=0.001s \ + --benchmark_out=parquet-kernel-smoke.json \ + --benchmark_out_format=json + be/output/lib/benchmark_test \ --benchmark_filter='^ParquetReader/' \ --benchmark_min_time=0.001s \ @@ -98,8 +108,8 @@ cache to manufacture a cold run. ## Current scenario matrix -`ParquetDecoder` contains 19 encoding/type pairs. Each pair is run at 1%, 10%, 50%, and 100% -selection with clustered and alternating selection ranges, for 152 registered cases. +`ParquetDecoder` contains 19 encoding/type pairs. Each pair is run at 0%, 1%, 10%, 50%, 90%, and +100% selection with clustered and alternating selection ranges, for 228 registered cases. | Encoding | Physical types | |---|---| @@ -110,10 +120,16 @@ selection with clustered and alternating selection ranges, for 152 registered ca | DELTA_LENGTH_BYTE_ARRAY | BYTE_ARRAY | | DELTA_BYTE_ARRAY | BYTE_ARRAY | +`ParquetKernel` contains 80 cases across five SIMD-sensitive stages: BYTE_STREAM_SPLIT, +DELTA_PREFIX_SUM, DICTIONARY_GATHER, NULLABLE_EXPAND, and RAW_PREDICATE. It covers the applicable +four- and eight-byte types, three dictionary working-set sizes, 0% through 90% null rates with both +placement patterns, and 0% through 100% raw-predicate selectivities. + `ParquetReader` deliberately uses a single-variable matrix rather than a Cartesian product. After -deduplication it contains 137 cases covering: +deduplication it contains 152 cases covering: -- operations: open-to-first-block, full scan, predicate scan, limit 1, and limit 1000; +- operations: open-to-first-block, full scan, predicate scan, complex residual scan, limit 1, and + limit 1000; - file encodings: PLAIN, dictionary, BYTE_STREAM_SPLIT, and DELTA_BINARY_PACKED; - null ratios: 0%, 1%, 10%, 50%, and 90%; - null shapes: clustered and alternating; @@ -146,10 +162,12 @@ generator, random seed, or manifest involved. and intentionally produces many one-row physical ranges. Page generation, selection construction, decoder creation, dictionary setup, and `set_data` are -outside the timed decode call. The sinks consume decoder callbacks and prevent compiler removal, -but they do not build a Doris `Column`. Consequently these cases isolate decoder traversal and -selection cost; they do not measure definition-level decoding, nullable reconstruction, type -conversion, or full column materialization. +outside the timed decode call. Before timing, every decoder case verifies the consumed value count +and a checksum of all selected values against the deterministic source generator. The timed sinks +then consume decoder callbacks and prevent compiler removal, but they do not build a Doris +`Column`. Consequently these cases isolate decoder traversal and selection cost; they do not +measure definition-level decoding, nullable reconstruction, type conversion, or full column +materialization. ## How reader Parquet files are generated @@ -167,6 +185,9 @@ Fixture contents and writer settings are: - every non-null value is `row % 100`; - the predicate is `value < selectivity_percent`, so the threshold maps directly to the intended non-null selectivity; +- the complex residual scan evaluates a production expression tree whose first child is + `c0 < selectivity_percent` and whose always-true second child is `c2 = c3`, exposing whether + later-only columns are decoded eagerly; - alternating nulls use a 101-row period and `(row * 37) % 101`, avoiding direct correlation with the 100-value predicate period; - clustered nulls use contiguous null prefixes inside each 1,024-row cluster; @@ -234,13 +255,14 @@ The current matrix is not comprehensive. Preserve this distinction in PR descrip 1. Add the design's `matrix.yaml`, deterministic corpus generator, `manifest.json`, checksum, and standalone corpus verifier. The current runtime-generated files cannot be shared unchanged with V1, StarRocks, or DuckDB. -2. Add correctness oracles. Benchmarks must validate consumed counts and representative checksums - outside the timed region before their performance samples are trusted. +2. Add full reader correctness oracles. Decoder cases validate consumed counts and selected-value + checksums outside the timed region, and kernel cases compare representative output. Reader cases + still need value checksums in addition to their output-row counters. 3. Add reader-level INT64, FLOAT, DOUBLE, BYTE_ARRAY/string, FIXED_LEN_BYTE_ARRAY, DATE, TIMESTAMP, and DECIMAL cases. Today only nullable INT32 reaches the complete reader path. -4. Extend decoder coverage with 0% and 90% selection, definition levels/null reconstruction, - dictionary conversion, and real Doris `Column` materialization. The current decoder sink does - not cover those costs or report decoded bytes per second. +4. Extend decoder coverage with definition levels/null reconstruction, dictionary conversion, and + real Doris `Column` materialization. The current decoder sink does not cover those costs or + report decoded bytes per second. 5. Add the representative nullable sparse corpus requested by the design: 32 INT64 columns, 128 row groups, and enough rows to exercise many pages. The current 16K-row/four-row-group fixture is a smoke-sized workload. @@ -277,10 +299,10 @@ be simulated by silently changing the local reader benchmark. ## Current validation record -At commit `16e05dd5c71`, a Release build completed and the matrix unit test passed 4/4. A 1 ms smoke -run executed 152 decoder and 137 reader cases with zero benchmark errors. This is an execution -record only. It is not a reviewed performance baseline because repetitions, host isolation, -warmups, cache control, `perf` data, variance, and before/after comparison were not collected. +The current expected registration counts are 228 decoder, 80 kernel, and 152 reader cases. A smoke +run is an execution record only, not a reviewed performance baseline, because repetitions, host +isolation, warmups, cache control, `perf` data, variance, and before/after comparison are not +collected. ## Rules for extending the suite diff --git a/be/benchmark/parquet/README.md b/be/benchmark/parquet/README.md index 40c79bc28bea5f..e6eab6362ce29d 100644 --- a/be/benchmark/parquet/README.md +++ b/be/benchmark/parquet/README.md @@ -28,21 +28,41 @@ timed region. It covers PLAIN, dictionary, byte-stream-split, and DELTA encoding supported fixed-width and binary physical types. Sparse selections are provided as both one clustered range and many alternating ranges. +The decoder selection axis includes 0%, 1%, 10%, 50%, 90%, and 100% so boundary and +high-selectivity behavior are visible. + ```shell be/output/lib/benchmark_test \ --benchmark_filter='^ParquetDecoder/plain/int64/sel_10/alternating$' \ --benchmark_min_time=0.1s ``` +## SIMD kernel cases + +`ParquetKernel` isolates the five SIMD-sensitive stages from reader setup and virtual consumer +overhead: byte-stream-split transpose, delta prefix sum, numeric dictionary gather, nullable +expansion, and raw predicate evaluation. It covers the applicable 4-byte and 8-byte integer and +floating-point physical types, raw-predicate selectivities from 0% through 100%, and nullable +rates from 0% through 90% with clustered and alternating placement. Dictionary gather uses 32-, +4,096-, and 262,144-entry working sets to separate cache-resident and cache-miss-dominated +behavior. + +```shell +be/output/lib/benchmark_test \ + --benchmark_filter='^ParquetKernel/(dictionary_gather|nullable_expand)/' \ + --benchmark_min_time=0.1s +``` + ## Local reader cases -`ParquetReader` measures local open-to-first-block, full scan, predicate scan, and LIMIT-shaped -reads. The matrix covers: +`ParquetReader` measures local open-to-first-block, full scan, predicate scan, complex residual +scan, and LIMIT-shaped reads. The matrix covers: - PLAIN, dictionary, byte-stream-split, and DELTA binary-packed files; - NULL ratios of 0%, 1%, 10%, 50%, and 90%, with clustered and alternating placement; - predicate selectivities of 0%, 1%, 10%, 50%, 90%, and 100%; - predicate-only and predicate-plus-lazy-projected reads; +- ordered complex residuals whose later columns are reachable only after an earlier residual; - schemas with 4, 32, 128, and 512 columns, with the predicate first or last. Fixtures are created lazily under the system temporary directory in @@ -58,6 +78,16 @@ be/output/lib/benchmark_test \ --benchmark_out_format=json ``` +The complex-residual case uses a production compound `AND` tree. Its first child, +`c0 < selectivity_percent`, preserves the requested selectivity; its second child, `c2 = c3`, +references two new columns and accepts every row that reaches it: + +```shell +be/output/lib/benchmark_test \ + --benchmark_filter='^ParquetReader/complex_residual_scan/plain/null_10/alternating/sel_10/' \ + --benchmark_min_time=1s +``` + Every result reports throughput plus `raw_rows`, `selected_rows`, `fixture_bytes`, `ns/raw_row`, and (when at least one row survives) `ns/selected_row`. Keep CPU frequency, build type, compiler, machine placement, and benchmark filters fixed when comparing two commits. diff --git a/be/benchmark/parquet/benchmark_parquet_decoder.hpp b/be/benchmark/parquet/benchmark_parquet_decoder.hpp index 25fb05d1449e53..c1c2e3cb33220c 100644 --- a/be/benchmark/parquet/benchmark_parquet_decoder.hpp +++ b/be/benchmark/parquet/benchmark_parquet_decoder.hpp @@ -101,31 +101,87 @@ inline std::shared_ptr<::parquet::ColumnDescriptor> descriptor(::parquet::Type:: return std::make_shared<::parquet::ColumnDescriptor>(node, 0, 0); } +template +T fixed_value(size_t row) { + if constexpr (std::is_floating_point_v) { + return static_cast((row % 1009) * 0.25 - 100.0); + } + return static_cast((row * 17) % 1000003); +} + template std::vector fixed_values(size_t rows) { std::vector values(rows); for (size_t row = 0; row < rows; ++row) { - if constexpr (std::is_floating_point_v) { - values[row] = static_cast((row % 1009) * 0.25 - 100.0); - } else { - values[row] = static_cast((row * 17) % 1000003); - } + values[row] = fixed_value(row); } return values; } +inline std::string binary_value(size_t row, size_t width = FIXED_BINARY_WIDTH) { + std::string value(width, 'a'); + const uint64_t id = row % 1009; + memcpy(value.data(), &id, std::min(width, sizeof(id))); + return value; +} + inline std::vector binary_values(size_t rows, size_t width = FIXED_BINARY_WIDTH) { std::vector values; values.reserve(rows); for (size_t row = 0; row < rows; ++row) { - std::string value(width, 'a'); - const uint64_t id = row % 1009; - memcpy(value.data(), &id, std::min(width, sizeof(id))); - values.push_back(std::move(value)); + values.push_back(binary_value(row, width)); } return values; } +struct DecoderDigest { + size_t consumed = 0; + uint64_t checksum = 1469598103934665603ULL; +}; + +inline void add_digest_value(DecoderDigest* digest, const uint8_t* bytes, size_t size) { + constexpr uint64_t FNV_PRIME = 1099511628211ULL; + digest->checksum ^= size; + digest->checksum *= FNV_PRIME; + for (size_t byte = 0; byte < size; ++byte) { + digest->checksum ^= bytes[byte]; + digest->checksum *= FNV_PRIME; + } + ++digest->consumed; +} + +inline void add_generated_value(DecoderDigest* digest, ValueType value_type, size_t row) { + switch (value_type) { + case ValueType::INT32: { + const auto value = fixed_value(row); + add_digest_value(digest, reinterpret_cast(&value), sizeof(value)); + return; + } + case ValueType::INT64: { + const auto value = fixed_value(row); + add_digest_value(digest, reinterpret_cast(&value), sizeof(value)); + return; + } + case ValueType::FLOAT: { + const auto value = fixed_value(row); + add_digest_value(digest, reinterpret_cast(&value), sizeof(value)); + return; + } + case ValueType::DOUBLE: { + const auto value = fixed_value(row); + add_digest_value(digest, reinterpret_cast(&value), sizeof(value)); + return; + } + case ValueType::BYTE_ARRAY: + case ValueType::FIXED_LEN_BYTE_ARRAY: { + const auto value = binary_value(row); + add_digest_value(digest, reinterpret_cast(value.data()), value.size()); + return; + } + } + throw std::logic_error("unknown Parquet benchmark value type"); +} + inline std::vector encode_plain_binary(const std::vector& values) { size_t bytes = 0; for (const auto& value : values) { @@ -376,6 +432,67 @@ class DictionarySink final : public ParquetDictionaryValueConsumer { const size_t _value_width; }; +class FixedVerificationSink final : public ParquetFixedValueConsumer { +public: + Status consume(const uint8_t* values, size_t num_values, size_t value_width) override { + for (size_t row = 0; row < num_values; ++row) { + add_digest_value(&digest, values + row * value_width, value_width); + } + return Status::OK(); + } + + DecoderDigest digest; +}; + +class BinaryVerificationSink final : public ParquetBinaryValueConsumer { +public: + Status consume(const StringRef* values, size_t num_values) override { + for (size_t row = 0; row < num_values; ++row) { + add_digest_value(&digest, reinterpret_cast(values[row].data), + values[row].size); + } + return Status::OK(); + } + + Status consume_plain_byte_array( + const char* encoded_data, const uint32_t* payload_offsets, + const uint32_t* value_offsets, size_t num_values, + const std::vector& value_spans) override { + for (size_t row = 0; row < num_values; ++row) { + add_digest_value(&digest, + reinterpret_cast(encoded_data + payload_offsets[row]), + value_offsets[row + 1] - value_offsets[row]); + } + return Status::OK(); + } + + DecoderDigest digest; +}; + +class DictionaryVerificationSink final : public ParquetDictionaryValueConsumer { +public: + explicit DictionaryVerificationSink(ValueType value_type) : _value_type(value_type) {} + + Status consume_indices(const uint32_t* indices, size_t num_values) override { + for (size_t row = 0; row < num_values; ++row) { + add_generated_value(&digest, _value_type, indices[row]); + } + return Status::OK(); + } + + Status consume_repeated(uint32_t index, size_t num_values) override { + for (size_t row = 0; row < num_values; ++row) { + add_generated_value(&digest, _value_type, index); + } + return Status::OK(); + } + + DecoderDigest digest; + +private: + const ValueType _value_type; +}; + inline ParquetSelection native_selection(const SelectionPlan& plan) { ParquetSelection selection { .total_values = plan.total_rows, .selected_values = plan.selected_rows, .ranges = {}}; @@ -386,6 +503,50 @@ inline ParquetSelection native_selection(const SelectionPlan& plan) { return selection; } +inline DecoderDigest expected_decoder_digest(const DecoderScenario& scenario, + const SelectionPlan& plan) { + DecoderDigest digest; + visit_selected_rows(plan, [&](size_t row) { + const size_t value_row = + scenario.encoding == Encoding::DICTIONARY ? row % DICTIONARY_ENTRIES : row; + add_generated_value(&digest, scenario.value_type, value_row); + }); + return digest; +} + +inline Status verify_decoder_output(format::parquet::native::Decoder* decoder, Slice* encoded, + const DecoderScenario& scenario, bool binary, + const ParquetSelection& selection, + const SelectionPlan& plan) { + RETURN_IF_ERROR(decoder->set_data(encoded)); + DecoderDigest actual; + if (scenario.encoding == Encoding::DICTIONARY) { + DictionaryVerificationSink sink(scenario.value_type); + RETURN_IF_ERROR(decoder->decode_selected_dictionary_values(selection, sink)); + actual = sink.digest; + } else if (binary) { + BinaryVerificationSink sink; + RETURN_IF_ERROR(decoder->decode_selected_binary_values(selection, sink)); + actual = sink.digest; + } else { + FixedVerificationSink sink; + RETURN_IF_ERROR(decoder->decode_selected_fixed_values(selection, sink)); + actual = sink.digest; + } + + const auto expected = expected_decoder_digest(scenario, plan); + // Validate sparse/boundary selections before timing; counters alone can hide wrong selected + // rows while still producing a plausible benchmark result. + if (actual.consumed != plan.selected_rows || actual.consumed != expected.consumed || + actual.checksum != expected.checksum) { + return Status::InternalError( + "Parquet decoder benchmark oracle mismatch: consumed {} expected {}, checksum {} " + "expected {}", + actual.consumed, plan.selected_rows, actual.checksum, expected.checksum); + } + return Status::OK(); +} + inline void run_decoder(benchmark::State& state, DecoderScenario scenario, int selectivity, Pattern pattern) { auto page = encoded_page(scenario); @@ -415,6 +576,12 @@ inline void run_decoder(benchmark::State& state, DecoderScenario scenario, int s } Slice encoded(page.data.data(), page.data.size()); + status = verify_decoder_output(decoder.get(), &encoded, scenario, page.binary, selection, plan); + if (!status.ok()) { + state.SkipWithError(status.to_string().c_str()); + return; + } + FixedSink fixed_sink; BinarySink binary_sink; DictionarySink dictionary_sink(dictionary_for_sink.data(), page.value_width); @@ -459,7 +626,7 @@ inline void run_decoder(benchmark::State& state, DecoderScenario scenario, int s inline bool register_decoder_benchmarks() { for (const auto& scenario : decoder_scenarios()) { - for (const int selectivity : {1, 10, 50, 100}) { + for (const int selectivity : {0, 1, 10, 50, 90, 100}) { for (const auto pattern : {Pattern::CLUSTERED, Pattern::ALTERNATING}) { const std::string name = "ParquetDecoder/" + to_string(scenario.encoding) + "/" + to_string(scenario.value_type) + "/sel_" + diff --git a/be/benchmark/parquet/benchmark_parquet_kernels.hpp b/be/benchmark/parquet/benchmark_parquet_kernels.hpp new file mode 100644 index 00000000000000..e064e69f64fdb9 --- /dev/null +++ b/be/benchmark/parquet/benchmark_parquet_kernels.hpp @@ -0,0 +1,243 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +#pragma once + +#include + +#include +#include +#include +#include +#include +#include + +#include "parquet_benchmark_scenarios.h" +#include "util/byte_stream_split.h" +#include "util/simd/parquet_kernels.h" + +namespace doris::parquet_benchmark { +namespace detail { + +constexpr size_t KERNEL_ROWS = 1UL << 16; + +inline void decode_byte_stream_split(const uint8_t* src, size_t width, size_t offset, + size_t num_values, size_t stride, uint8_t* dest) { + if (!simd::try_byte_stream_split_decode(src, width, offset, num_values, stride, dest)) { + doris::byte_stream_split_decode(src, static_cast(width), offset, num_values, stride, + dest); + } +} + +template +void run_kernel(benchmark::State& state, const KernelScenario& scenario) { + constexpr size_t width = sizeof(T); + std::vector input(KERNEL_ROWS); + for (size_t row = 0; row < input.size(); ++row) { + input[row] = scenario.kernel == Kernel::RAW_PREDICATE ? static_cast(row % 100) + : static_cast((row * 17) % 1009); + } + std::vector output(KERNEL_ROWS); + + std::vector encoded(KERNEL_ROWS * width); + for (size_t row = 0; row < KERNEL_ROWS; ++row) { + for (size_t byte = 0; byte < width; ++byte) { + encoded[byte * KERNEL_ROWS + row] = + reinterpret_cast(input.data())[row * width + byte]; + } + } + + std::vector dictionary(scenario.dictionary_entries); + for (size_t row = 0; row < dictionary.size(); ++row) { + dictionary[row] = static_cast(row * 31); + } + std::vector ids(KERNEL_ROWS); + for (size_t row = 0; row < ids.size(); ++row) { + ids[row] = static_cast((row * 13) % dictionary.size()); + } + + std::vector nulls(KERNEL_ROWS, 0); + const size_t null_count = KERNEL_ROWS * static_cast(scenario.null_percent) / 100; + if (scenario.pattern == Pattern::CLUSTERED) { + std::fill_n(nulls.begin(), null_count, uint8_t {1}); + } else if (null_count != 0) { + for (size_t value = 0; value < null_count; ++value) { + nulls[value * KERNEL_ROWS / null_count] = 1; + } + } + std::vector compact; + compact.reserve(KERNEL_ROWS - null_count); + for (size_t row = 0; row < KERNEL_ROWS; ++row) { + if (nulls[row] == 0) { + compact.push_back(input[row]); + } + } + + const T literal = static_cast(scenario.selectivity_percent); + std::vector matches(KERNEL_ROWS, 1); + + switch (scenario.kernel) { + case Kernel::BYTE_STREAM_SPLIT: + decode_byte_stream_split(encoded.data(), width, 0, KERNEL_ROWS, KERNEL_ROWS, + reinterpret_cast(output.data())); + if (output != input) { + state.SkipWithError("byte-stream-split kernel produced incorrect values"); + return; + } + break; + case Kernel::DELTA_PREFIX_SUM: { + if constexpr (std::is_integral_v) { + std::copy(input.begin(), input.end(), output.begin()); + auto expected = output; + T expected_last = 7; + using Unsigned = std::make_unsigned_t; + for (auto& value : expected) { + value = static_cast(static_cast(value) + + static_cast(static_cast(-3)) + + static_cast(expected_last)); + expected_last = value; + } + T last = 7; + simd::delta_decode(output.data(), output.size(), static_cast(-3), &last); + if (output != expected || last != expected_last) { + state.SkipWithError("delta prefix-sum kernel produced incorrect values"); + return; + } + } else { + state.SkipWithError("delta prefix-sum requires an integer physical type"); + return; + } + break; + } + case Kernel::DICTIONARY_GATHER: + simd::dictionary_gather(reinterpret_cast(dictionary.data()), ids.data(), + ids.size(), width, reinterpret_cast(output.data())); + for (size_t row = 0; row < output.size(); ++row) { + if (output[row] != dictionary[ids[row]]) { + state.SkipWithError("dictionary gather kernel produced incorrect values"); + return; + } + } + break; + case Kernel::NULLABLE_EXPAND: + std::copy(compact.begin(), compact.end(), output.begin()); + simd::expand_nullable_values(reinterpret_cast(output.data()), compact.size(), + nulls.data(), nulls.size(), width); + for (size_t row = 0; row < output.size(); ++row) { + const T expected = nulls[row] == 0 ? input[row] : T {}; + if (output[row] != expected) { + state.SkipWithError("nullable expansion kernel produced incorrect values"); + return; + } + } + break; + case Kernel::RAW_PREDICATE: + simd::raw_compare(reinterpret_cast(input.data()), input.size(), literal, + simd::RawComparisonOp::LT, matches.data()); + for (size_t row = 0; row < matches.size(); ++row) { + if (matches[row] != static_cast(input[row] < literal)) { + state.SkipWithError("raw predicate kernel produced incorrect values"); + return; + } + } + break; + } + + for (auto _ : state) { + state.PauseTiming(); + if (scenario.kernel == Kernel::DELTA_PREFIX_SUM) { + std::copy(input.begin(), input.end(), output.begin()); + } else if (scenario.kernel == Kernel::NULLABLE_EXPAND) { + std::copy(compact.begin(), compact.end(), output.begin()); + } else if (scenario.kernel == Kernel::RAW_PREDICATE) { + std::fill(matches.begin(), matches.end(), uint8_t {1}); + } + state.ResumeTiming(); + + switch (scenario.kernel) { + case Kernel::BYTE_STREAM_SPLIT: + decode_byte_stream_split(encoded.data(), width, 0, KERNEL_ROWS, KERNEL_ROWS, + reinterpret_cast(output.data())); + break; + case Kernel::DELTA_PREFIX_SUM: { + if constexpr (std::is_integral_v) { + T last = 7; + simd::delta_decode(output.data(), output.size(), static_cast(-3), &last); + benchmark::DoNotOptimize(last); + } + break; + } + case Kernel::DICTIONARY_GATHER: + simd::dictionary_gather(reinterpret_cast(dictionary.data()), ids.data(), + ids.size(), width, reinterpret_cast(output.data())); + break; + case Kernel::NULLABLE_EXPAND: + simd::expand_nullable_values(reinterpret_cast(output.data()), compact.size(), + nulls.data(), nulls.size(), width); + break; + case Kernel::RAW_PREDICATE: + simd::raw_compare(reinterpret_cast(input.data()), input.size(), literal, + simd::RawComparisonOp::LT, matches.data()); + break; + } + benchmark::ClobberMemory(); + } + + state.SetItemsProcessed(static_cast(state.iterations()) * + static_cast(KERNEL_ROWS)); + state.SetBytesProcessed(static_cast(state.iterations()) * + static_cast(KERNEL_ROWS * width)); + state.counters["rows"] = static_cast(KERNEL_ROWS); + state.counters["value_width"] = static_cast(width); +} + +inline bool register_kernel_benchmarks() { + for (const auto& scenario : kernel_scenarios()) { + const std::string name = "ParquetKernel/" + to_string(scenario.kernel) + "/" + + to_string(scenario.value_type) + "/sel_" + + std::to_string(scenario.selectivity_percent) + "/null_" + + std::to_string(scenario.null_percent) + "/" + + to_string(scenario.pattern) + "/dict_" + + std::to_string(scenario.dictionary_entries); + benchmark::RegisterBenchmark(name.c_str(), [=](benchmark::State& state) { + switch (scenario.value_type) { + case ValueType::INT32: + run_kernel(state, scenario); + break; + case ValueType::INT64: + run_kernel(state, scenario); + break; + case ValueType::FLOAT: + run_kernel(state, scenario); + break; + case ValueType::DOUBLE: + run_kernel(state, scenario); + break; + case ValueType::BYTE_ARRAY: + case ValueType::FIXED_LEN_BYTE_ARRAY: + state.SkipWithError("kernel benchmark requires a fixed-width primitive type"); + break; + } + })->Unit(benchmark::kNanosecond); + } + return true; +} + +inline const bool KERNEL_BENCHMARKS_REGISTERED = register_kernel_benchmarks(); + +} // namespace detail +} // namespace doris::parquet_benchmark diff --git a/be/benchmark/parquet/benchmark_parquet_reader.hpp b/be/benchmark/parquet/benchmark_parquet_reader.hpp index ae3e14c0f66ad9..b456244ba0e3ba 100644 --- a/be/benchmark/parquet/benchmark_parquet_reader.hpp +++ b/be/benchmark/parquet/benchmark_parquet_reader.hpp @@ -24,6 +24,7 @@ #include #include +#include #include #include #include @@ -39,13 +40,18 @@ #include "core/column/column_vector.h" #include "core/data_type/data_type_nullable.h" #include "core/data_type/data_type_number.h" +#include "exprs/vcompound_pred.h" +#include "exprs/vectorized_fn_call.h" #include "exprs/vexpr.h" #include "exprs/vexpr_context.h" +#include "exprs/vliteral.h" +#include "exprs/vslot_ref.h" #include "format_v2/file_reader.h" #include "format_v2/parquet/parquet_reader.h" #include "gen_cpp/Types_types.h" #include "io/io_common.h" #include "parquet_benchmark_scenarios.h" +#include "runtime/descriptors.h" #include "runtime/runtime_state.h" #include "storage/index/zone_map/zonemap_eval_context.h" #include "storage/index/zone_map/zonemap_filter_result.h" @@ -88,6 +94,64 @@ inline std::shared_ptr build_int32_array(int null_percent, Pattern return builder.Finish().ValueOrDie(); } +inline std::shared_ptr build_int64_array(int null_percent, Pattern pattern) { + arrow::Int64Builder builder; + PARQUET_THROW_NOT_OK(builder.Reserve(READER_ROWS)); + for (size_t row = 0; row < READER_ROWS; ++row) { + if (is_null_row(row, null_percent, pattern)) { + PARQUET_THROW_NOT_OK(builder.AppendNull()); + } else { + PARQUET_THROW_NOT_OK(builder.Append(static_cast(row % 100))); + } + } + return builder.Finish().ValueOrDie(); +} + +inline std::string padded_decimal(size_t value) { + std::string result = std::to_string(value); + result.insert(0, 3 - result.size(), '0'); + return result; +} + +inline std::shared_ptr build_string_array(int null_percent, Pattern pattern) { + arrow::StringBuilder builder; + PARQUET_THROW_NOT_OK(builder.Reserve(READER_ROWS)); + for (size_t row = 0; row < READER_ROWS; ++row) { + if (is_null_row(row, null_percent, pattern)) { + PARQUET_THROW_NOT_OK(builder.AppendNull()); + } else { + PARQUET_THROW_NOT_OK(builder.Append(padded_decimal(row % 100))); + } + } + return builder.Finish().ValueOrDie(); +} + +inline std::shared_ptr build_value_array(const ReaderScenario& scenario) { + switch (scenario.value_type) { + case ValueType::INT32: + return build_int32_array(scenario.null_percent, scenario.null_pattern); + case ValueType::INT64: + return build_int64_array(scenario.null_percent, scenario.null_pattern); + case ValueType::BYTE_ARRAY: + return build_string_array(scenario.null_percent, scenario.null_pattern); + default: + throw std::logic_error("unsupported Parquet reader benchmark value type"); + } +} + +inline std::shared_ptr arrow_value_type(ValueType value_type) { + switch (value_type) { + case ValueType::INT32: + return arrow::int32(); + case ValueType::INT64: + return arrow::int64(); + case ValueType::BYTE_ARRAY: + return arrow::utf8(); + default: + throw std::logic_error("unsupported Parquet reader benchmark value type"); + } +} + inline ::parquet::Encoding::type file_encoding(Encoding encoding) { switch (encoding) { case Encoding::PLAIN: @@ -107,9 +171,10 @@ inline ::parquet::Encoding::type file_encoding(Encoding encoding) { } inline std::string fixture_name(const ReaderScenario& scenario) { - return "v2_" + to_string(scenario.encoding) + "_null" + std::to_string(scenario.null_percent) + - "_" + to_string(scenario.null_pattern) + "_w" + std::to_string(scenario.schema_width) + - "_p" + std::to_string(scenario.predicate_position) + ".parquet"; + return "v2_" + to_string(scenario.encoding) + "_" + to_string(scenario.value_type) + "_null" + + std::to_string(scenario.null_percent) + "_" + to_string(scenario.null_pattern) + "_w" + + std::to_string(scenario.schema_width) + "_p" + + std::to_string(scenario.predicate_position) + ".parquet"; } inline void verify_fixture_encoding(const std::filesystem::path& path, @@ -148,13 +213,14 @@ inline std::filesystem::path ensure_fixture(const ReaderScenario& scenario) { std::filesystem::create_directories(directory); const auto temporary_path = path.string() + ".tmp"; std::filesystem::remove(temporary_path); - const auto values = build_int32_array(scenario.null_percent, scenario.null_pattern); + const auto values = build_value_array(scenario); std::vector> fields; std::vector> columns; fields.reserve(scenario.schema_width); columns.reserve(scenario.schema_width); for (int column = 0; column < scenario.schema_width; ++column) { - fields.push_back(arrow::field("c" + std::to_string(column), arrow::int32(), true)); + fields.push_back(arrow::field("c" + std::to_string(column), + arrow_value_type(scenario.value_type), true)); columns.push_back(std::make_shared(values)); } const auto table = arrow::Table::Make(arrow::schema(std::move(fields)), std::move(columns)); @@ -275,6 +341,73 @@ inline VExprContextSPtr make_predicate(int column_position, int selectivity_perc return context; } +inline VExprSPtr make_int32_comparison(const std::string& function_name, TExprOpcode::type opcode, + VExprSPtr left, VExprSPtr right) { + const auto bool_type = make_nullable(std::make_shared()); + TFunctionName name; + name.__set_function_name(function_name); + TFunction function; + function.__set_name(name); + function.__set_binary_type(TFunctionBinaryType::BUILTIN); + function.__set_arg_types({left->data_type()->to_thrift(), right->data_type()->to_thrift()}); + function.__set_ret_type(bool_type->to_thrift()); + function.__set_has_var_args(false); + TExprNode node; + node.__set_node_type(TExprNodeType::BINARY_PRED); + node.__set_opcode(opcode); + node.__set_type(bool_type->to_thrift()); + node.__set_fn(function); + node.__set_num_children(2); + node.__set_is_nullable(true); + auto comparison = VectorizedFnCall::create_shared(node); + comparison->add_child(std::move(left)); + comparison->add_child(std::move(right)); + return comparison; +} + +inline VExprSPtr make_reader_literal(const ReaderScenario& scenario, const DataTypePtr& type) { + switch (scenario.value_type) { + case ValueType::INT32: + return VLiteral::create_shared(remove_nullable(type), + Field::create_field(scenario.selectivity_percent)); + case ValueType::INT64: + return VLiteral::create_shared( + remove_nullable(type), + Field::create_field(scenario.selectivity_percent)); + case ValueType::BYTE_ARRAY: + return VLiteral::create_shared( + remove_nullable(type), + Field::create_field(padded_decimal(scenario.selectivity_percent))); + default: + throw std::logic_error("unsupported Parquet reader benchmark predicate type"); + } +} + +inline VExprContextSPtr make_complex_residual_predicate(int selectivity_percent, int first_position, + int later_left_position, + int later_right_position, + const DataTypePtr& int_type) { + const auto bool_type = make_nullable(std::make_shared()); + TExprNode node; + node.__set_node_type(TExprNodeType::COMPOUND_PRED); + node.__set_opcode(TExprOpcode::COMPOUND_AND); + node.__set_type(bool_type->to_thrift()); + node.__set_num_children(2); + node.__set_is_nullable(true); + auto compound = VCompoundPred::create_shared(node); + compound->add_child(make_int32_comparison( + "lt", TExprOpcode::LT, + VSlotRef::create_shared(first_position, first_position, -1, int_type, "c0"), + VLiteral::create_shared(remove_nullable(int_type), + Field::create_field(selectivity_percent)))); + compound->add_child(make_int32_comparison( + "eq", TExprOpcode::EQ, + VSlotRef::create_shared(later_left_position, later_left_position, -1, int_type, "c2"), + VSlotRef::create_shared(later_right_position, later_right_position, -1, int_type, + "c3"))); + return VExprContext::create_shared(std::move(compound)); +} + inline Block make_block(const std::vector& schema) { Block block; for (const auto& column : schema) { @@ -284,10 +417,17 @@ inline Block make_block(const std::vector& schema) { } struct ReaderSession { + ~ReaderSession() { + for (const auto& context : opened_conjuncts) { + context->close(); + } + } + RuntimeState runtime_state {TQueryOptions(), TQueryGlobals()}; std::unique_ptr reader; std::vector schema; std::shared_ptr request; + VExprContextSPtrs opened_conjuncts; }; inline std::unique_ptr open_reader(const std::filesystem::path& path, @@ -322,8 +462,37 @@ inline std::unique_ptr open_reader(const std::filesystem::path& p request_builder.add_non_predicate_column(format::LocalColumnId(payload))); } const auto predicate_position = session->request->local_positions.at(predicate_id).value(); - session->request->conjuncts.push_back( - make_predicate(static_cast(predicate_position), scenario.selectivity_percent)); + auto context = VExprContext::create_shared(make_int32_comparison( + "lt", TExprOpcode::LT, + VSlotRef::create_shared(static_cast(predicate_position), + static_cast(predicate_position), -1, + session->schema[scenario.predicate_position].type, "c0"), + make_reader_literal(scenario, session->schema[scenario.predicate_position].type))); + throw_if_error(context->prepare(&session->runtime_state, RowDescriptor())); + throw_if_error(context->open(&session->runtime_state)); + session->request->conjuncts.push_back(context); + session->opened_conjuncts.push_back(std::move(context)); + } else if (scenario.operation == ReaderOperation::COMPLEX_RESIDUAL_SCAN) { + DORIS_CHECK(scenario.schema_width >= 5); + std::array predicate_columns {0, 2, 3}; + std::array predicate_positions {}; + for (size_t index = 0; index < predicate_columns.size(); ++index) { + const int column = predicate_columns[index]; + const auto predicate_id = format::LocalColumnId(column); + throw_if_error(request_builder.add_predicate_column(predicate_id)); + session->request->predicate_only_columns.push_back(predicate_id); + predicate_positions[index] = + static_cast(session->request->local_positions.at(predicate_id).value()); + } + throw_if_error(request_builder.add_non_predicate_column( + format::LocalColumnId(scenario.schema_width - 1))); + auto context = make_complex_residual_predicate( + scenario.selectivity_percent, predicate_positions[0], predicate_positions[1], + predicate_positions[2], session->schema[0].type); + throw_if_error(context->prepare(&session->runtime_state, RowDescriptor())); + throw_if_error(context->open(&session->runtime_state)); + session->request->conjuncts.push_back(context); + session->opened_conjuncts.push_back(std::move(context)); } else { throw_if_error(request_builder.add_non_predicate_column(format::LocalColumnId(0))); if (scenario.schema_width > 1) { @@ -383,9 +552,25 @@ inline int projected_columns(const ReaderScenario& scenario) { scenario.projection == Projection::PREDICATE_ONLY) { return 1; } + if (scenario.operation == ReaderOperation::COMPLEX_RESIDUAL_SCAN) { + return 4; + } return std::min(2, scenario.schema_width); } +inline size_t value_width(const ReaderScenario& scenario) { + switch (scenario.value_type) { + case ValueType::INT32: + return sizeof(int32_t); + case ValueType::INT64: + return sizeof(int64_t); + case ValueType::BYTE_ARRAY: + return 3; + default: + return sizeof(int32_t); + } +} + inline void run_reader(benchmark::State& state, ReaderScenario scenario) { std::filesystem::path fixture; try { @@ -408,8 +593,9 @@ inline void run_reader(benchmark::State& state, ReaderScenario scenario) { const auto raw_rows = raw_rows_per_iteration(scenario); state.SetItemsProcessed(static_cast(state.iterations() * selected_rows)); - state.SetBytesProcessed(static_cast( - state.iterations() * raw_rows * projected_columns(scenario) * sizeof(int32_t))); + state.SetBytesProcessed( + static_cast(state.iterations() * raw_rows * projected_columns(scenario) * + value_width(scenario))); state.counters["raw_rows"] = static_cast(raw_rows); state.counters["selected_rows"] = static_cast(selected_rows); state.counters["fixture_bytes"] = static_cast(std::filesystem::file_size(fixture)); @@ -428,13 +614,7 @@ inline void run_reader(benchmark::State& state, ReaderScenario scenario) { inline bool register_reader_benchmarks() { for (const auto& scenario : reader_scenarios()) { - std::string name = - "ParquetReader/" + to_string(scenario.operation) + "/" + - to_string(scenario.encoding) + "/null_" + std::to_string(scenario.null_percent) + - "/" + to_string(scenario.null_pattern) + "/sel_" + - std::to_string(scenario.selectivity_percent) + "/" + - to_string(scenario.projection) + "/width_" + std::to_string(scenario.schema_width) + - "/predicate_" + std::to_string(scenario.predicate_position); + std::string name = "ParquetReader/" + reader_scenario_name(scenario); benchmark::RegisterBenchmark(name.c_str(), [=](benchmark::State& state) { run_reader(state, scenario); })->Unit(benchmark::kNanosecond); diff --git a/be/benchmark/parquet/parquet_benchmark_scenarios.h b/be/benchmark/parquet/parquet_benchmark_scenarios.h index a01c955c6a2aed..900cb29a583478 100644 --- a/be/benchmark/parquet/parquet_benchmark_scenarios.h +++ b/be/benchmark/parquet/parquet_benchmark_scenarios.h @@ -37,7 +37,21 @@ enum class Encoding { enum class ValueType { INT32, INT64, FLOAT, DOUBLE, BYTE_ARRAY, FIXED_LEN_BYTE_ARRAY }; enum class Pattern { CLUSTERED, ALTERNATING }; enum class Projection { PREDICATE_ONLY, PREDICATE_PROJECTED }; -enum class ReaderOperation { OPEN_TO_FIRST_BLOCK, FULL_SCAN, PREDICATE_SCAN, LIMIT_1, LIMIT_1000 }; +enum class ReaderOperation { + OPEN_TO_FIRST_BLOCK, + FULL_SCAN, + PREDICATE_SCAN, + COMPLEX_RESIDUAL_SCAN, + LIMIT_1, + LIMIT_1000 +}; +enum class Kernel { + BYTE_STREAM_SPLIT, + DELTA_PREFIX_SUM, + DICTIONARY_GATHER, + NULLABLE_EXPAND, + RAW_PREDICATE +}; struct DecoderScenario { Encoding encoding; @@ -53,6 +67,16 @@ struct ReaderScenario { Projection projection; int schema_width; int predicate_position; + ValueType value_type = ValueType::INT32; +}; + +struct KernelScenario { + Kernel kernel; + ValueType value_type; + int selectivity_percent; + int null_percent; + Pattern pattern; + size_t dictionary_entries; }; struct SelectionRange { @@ -90,14 +114,46 @@ inline std::vector decoder_scenarios() { }; } +inline std::vector kernel_scenarios() { + std::vector scenarios; + for (const auto value_type : {ValueType::FLOAT, ValueType::DOUBLE}) { + scenarios.push_back( + {Kernel::BYTE_STREAM_SPLIT, value_type, 100, 0, Pattern::CLUSTERED, 256}); + } + for (const auto value_type : {ValueType::INT32, ValueType::INT64}) { + scenarios.push_back( + {Kernel::DELTA_PREFIX_SUM, value_type, 100, 0, Pattern::CLUSTERED, 256}); + } + for (const auto value_type : + {ValueType::INT32, ValueType::INT64, ValueType::FLOAT, ValueType::DOUBLE}) { + for (const size_t dictionary_entries : {32, 4096, 262144}) { + scenarios.push_back({Kernel::DICTIONARY_GATHER, value_type, 100, 0, Pattern::CLUSTERED, + dictionary_entries}); + } + for (const int null_percent : {0, 1, 10, 50, 90}) { + for (const auto pattern : {Pattern::CLUSTERED, Pattern::ALTERNATING}) { + scenarios.push_back( + {Kernel::NULLABLE_EXPAND, value_type, 100, null_percent, pattern, 256}); + } + } + for (const int selectivity : {0, 1, 10, 50, 90, 100}) { + scenarios.push_back( + {Kernel::RAW_PREDICATE, value_type, selectivity, 0, Pattern::ALTERNATING, 256}); + } + } + return scenarios; +} + inline std::vector reader_scenarios() { std::vector scenarios; - std::set> seen; + std::set> + seen; const auto add = [&](ReaderScenario scenario) { - const auto key = std::make_tuple(scenario.operation, scenario.encoding, - scenario.null_percent, scenario.null_pattern, - scenario.selectivity_percent, scenario.projection, - scenario.schema_width, scenario.predicate_position); + const auto key = std::make_tuple( + scenario.operation, scenario.encoding, scenario.null_percent, scenario.null_pattern, + scenario.selectivity_percent, scenario.projection, scenario.schema_width, + scenario.predicate_position, scenario.value_type); if (seen.insert(key).second) { scenarios.push_back(scenario); } @@ -113,7 +169,8 @@ inline std::vector reader_scenarios() { .predicate_position = 0}; for (const auto operation : {ReaderOperation::OPEN_TO_FIRST_BLOCK, ReaderOperation::FULL_SCAN, - ReaderOperation::PREDICATE_SCAN, ReaderOperation::LIMIT_1, ReaderOperation::LIMIT_1000}) { + ReaderOperation::PREDICATE_SCAN, ReaderOperation::COMPLEX_RESIDUAL_SCAN, + ReaderOperation::LIMIT_1, ReaderOperation::LIMIT_1000}) { auto scenario = baseline; scenario.operation = operation; add(scenario); @@ -139,6 +196,31 @@ inline std::vector reader_scenarios() { } } } + for (const int selectivity : {1, 10, 50, 90}) { + for (const auto projection : + {Projection::PREDICATE_ONLY, Projection::PREDICATE_PROJECTED}) { + auto scenario = baseline; + scenario.operation = ReaderOperation::PREDICATE_SCAN; + scenario.encoding = Encoding::DICTIONARY; + scenario.selectivity_percent = selectivity; + scenario.projection = projection; + add(scenario); + } + } + for (const auto value_type : {ValueType::INT64, ValueType::BYTE_ARRAY}) { + for (const int selectivity : {10, 50}) { + for (const auto projection : + {Projection::PREDICATE_ONLY, Projection::PREDICATE_PROJECTED}) { + auto scenario = baseline; + scenario.operation = ReaderOperation::PREDICATE_SCAN; + scenario.encoding = Encoding::DICTIONARY; + scenario.selectivity_percent = selectivity; + scenario.projection = projection; + scenario.value_type = value_type; + add(scenario); + } + } + } for (const int width : {4, 32, 128, 512}) { for (const int predicate_position : {0, width - 1}) { auto scenario = baseline; @@ -200,6 +282,15 @@ inline SelectionPlan make_selection_plan(size_t total_rows, int selectivity_perc return plan; } +template +inline void visit_selected_rows(const SelectionPlan& plan, Visitor visitor) { + for (const auto& range : plan.ranges) { + for (size_t offset = 0; offset < range.count; ++offset) { + visitor(range.first + offset); + } + } +} + inline std::string to_string(Encoding value) { switch (value) { case Encoding::PLAIN: @@ -252,6 +343,8 @@ inline std::string to_string(ReaderOperation value) { return "full_scan"; case ReaderOperation::PREDICATE_SCAN: return "predicate_scan"; + case ReaderOperation::COMPLEX_RESIDUAL_SCAN: + return "complex_residual_scan"; case ReaderOperation::LIMIT_1: return "limit_1"; case ReaderOperation::LIMIT_1000: @@ -260,4 +353,29 @@ inline std::string to_string(ReaderOperation value) { return "unknown"; } +inline std::string reader_scenario_name(const ReaderScenario& scenario) { + return to_string(scenario.operation) + "/" + to_string(scenario.encoding) + "/" + + to_string(scenario.value_type) + "/null_" + std::to_string(scenario.null_percent) + "/" + + to_string(scenario.null_pattern) + "/sel_" + + std::to_string(scenario.selectivity_percent) + "/" + to_string(scenario.projection) + + "/width_" + std::to_string(scenario.schema_width) + "/predicate_" + + std::to_string(scenario.predicate_position); +} + +inline std::string to_string(Kernel value) { + switch (value) { + case Kernel::BYTE_STREAM_SPLIT: + return "byte_stream_split"; + case Kernel::DELTA_PREFIX_SUM: + return "delta_prefix_sum"; + case Kernel::DICTIONARY_GATHER: + return "dictionary_gather"; + case Kernel::NULLABLE_EXPAND: + return "nullable_expand"; + case Kernel::RAW_PREDICATE: + return "raw_predicate"; + } + return "unknown"; +} + } // namespace doris::parquet_benchmark diff --git a/be/src/core/data_type_serde/parquet_decode_source.cpp b/be/src/core/data_type_serde/parquet_decode_source.cpp new file mode 100644 index 00000000000000..68df6074ddfa7a --- /dev/null +++ b/be/src/core/data_type_serde/parquet_decode_source.cpp @@ -0,0 +1,140 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +#include "core/data_type_serde/parquet_decode_source.h" + +#include +#include +#include + +#include "core/column/column_decimal.h" +#include "core/column/column_string.h" +#include "core/column/column_vector.h" +#include "util/simd/parquet_kernels.h" + +namespace doris { +namespace { + +template +bool try_gather_fixed_width(IColumn& destination, const IColumn& dictionary, + const uint32_t* indices, size_t num_values) { + using ValueType = typename ColumnType::value_type; + static_assert(std::is_trivially_copyable_v); + if constexpr (sizeof(ValueType) != 4 && sizeof(ValueType) != 8) { + return false; + } else { + auto* destination_vector = dynamic_cast(&destination); + const auto* dictionary_vector = dynamic_cast(&dictionary); + if (destination_vector == nullptr || dictionary_vector == nullptr) { + return false; + } + // The direct strategy is chosen only while the typed dictionary is cache-resident. Keep + // the existing generic insertion path for tiny batches where gather setup cannot amortize. + constexpr size_t SIMD_LANES = sizeof(ValueType) == 4 ? 8 : 4; + if (num_values < SIMD_LANES) { + return false; + } + auto& destination_data = destination_vector->get_data(); + const auto& dictionary_data = dictionary_vector->get_data(); + const size_t old_size = destination_data.size(); + destination_data.resize(old_size + num_values); + simd::dictionary_gather(reinterpret_cast(dictionary_data.data()), indices, + num_values, sizeof(ValueType), + reinterpret_cast(destination_data.data() + old_size)); + return true; + } +} + +template +bool try_gather_vector(IColumn& destination, const IColumn& dictionary, const uint32_t* indices, + size_t num_values) { + return try_gather_fixed_width>(destination, dictionary, indices, num_values); +} + +template +bool try_gather_strings(IColumn& destination, const IColumn& dictionary, const uint32_t* indices, + size_t num_values) { + auto* destination_string = dynamic_cast*>(&destination); + const auto* dictionary_string = dynamic_cast*>(&dictionary); + if (destination_string == nullptr || dictionary_string == nullptr) { + return false; + } + + size_t bytes = 0; + for (size_t row = 0; row < num_values; ++row) { + const size_t value_size = dictionary_string->get_data_at(indices[row]).size; + if (value_size > std::numeric_limits::max() - bytes) { + return false; + } + bytes += value_size; + } + auto& chars = destination_string->get_chars(); + auto& offsets = destination_string->get_offsets(); + if (bytes > std::numeric_limits::max() - chars.size()) { + return false; + } + const size_t old_chars_size = chars.size(); + chars.resize(old_chars_size + bytes); + offsets.reserve(offsets.size() + num_values); + size_t output_offset = old_chars_size; + for (size_t row = 0; row < num_values; ++row) { + const StringRef value = dictionary_string->get_data_at(indices[row]); + if (value.size != 0) { + memcpy(chars.data() + output_offset, value.data, value.size); + } + output_offset += value.size; + offsets.push_back(static_cast(output_offset)); + } + return true; +} + +} // namespace + +bool try_simd_insert_parquet_dictionary_indices(IColumn& destination, const IColumn& dictionary, + const uint32_t* indices, size_t num_values) { +#define TRY_PARQUET_GATHER(TYPE) \ + if (try_gather_vector(destination, dictionary, indices, num_values)) return true + TRY_PARQUET_GATHER(TYPE_INT); + TRY_PARQUET_GATHER(TYPE_BIGINT); + TRY_PARQUET_GATHER(TYPE_FLOAT); + TRY_PARQUET_GATHER(TYPE_DOUBLE); + TRY_PARQUET_GATHER(TYPE_DATE); + TRY_PARQUET_GATHER(TYPE_DATETIME); + TRY_PARQUET_GATHER(TYPE_DATEV2); + TRY_PARQUET_GATHER(TYPE_DATETIMEV2); + TRY_PARQUET_GATHER(TYPE_TIMESTAMPTZ); + TRY_PARQUET_GATHER(TYPE_IPV4); + TRY_PARQUET_GATHER(TYPE_TIMEV2); + TRY_PARQUET_GATHER(TYPE_UINT32); + TRY_PARQUET_GATHER(TYPE_UINT64); +#undef TRY_PARQUET_GATHER + // Keep every 4/8-byte POD column on this path: aliases such as TIMESTAMPTZ and decimals do not + // participate in the ordinary numeric dispatch and otherwise silently fall back to Field insertion. + if (try_gather_fixed_width(destination, dictionary, indices, num_values) || + try_gather_fixed_width(destination, dictionary, indices, num_values)) { + return true; + } + // String survivors have variable widths, so pre-size both buffers and copy each selected + // dictionary slice exactly once instead of routing every id through generic Field insertion. + if (try_gather_strings(destination, dictionary, indices, num_values) || + try_gather_strings(destination, dictionary, indices, num_values)) { + return true; + } + return false; +} + +} // namespace doris diff --git a/be/src/core/data_type_serde/parquet_decode_source.h b/be/src/core/data_type_serde/parquet_decode_source.h index e9762385689d07..18624abf45e7d7 100644 --- a/be/src/core/data_type_serde/parquet_decode_source.h +++ b/be/src/core/data_type_serde/parquet_decode_source.h @@ -253,6 +253,9 @@ class ParquetDecodeSource { enum class ParquetDictionaryMaterializationStrategy : uint8_t { DIRECT, INDICES }; +bool try_simd_insert_parquet_dictionary_indices(IColumn& destination, const IColumn& dictionary, + const uint32_t* indices, size_t num_values); + // Dictionary values are materialized once into the selected Doris type. The state belongs to a // column reader rather than DataTypeSerDe because a SerDe instance can be shared by many files. struct ParquetMaterializationState { @@ -363,7 +366,10 @@ struct ParquetMaterializationState { : _destination(destination), _dictionary(dictionary) {} Status consume_indices(const uint32_t* indices, size_t num_values) override { - _destination.insert_indices_from(_dictionary, indices, indices + num_values); + if (!try_simd_insert_parquet_dictionary_indices(_destination, _dictionary, indices, + num_values)) { + _destination.insert_indices_from(_dictionary, indices, indices + num_values); + } return Status::OK(); } diff --git a/be/src/exec/operator/aggregation_sink_operator.h b/be/src/exec/operator/aggregation_sink_operator.h index fe0a4023cdaabe..35823d075a2da1 100644 --- a/be/src/exec/operator/aggregation_sink_operator.h +++ b/be/src/exec/operator/aggregation_sink_operator.h @@ -160,6 +160,9 @@ class AggSinkOperatorX MOCK_REMOVE(final) : public DataSinkOperatorX::required_data_distribution( state); } + if (!_needs_finalize && !state->enable_local_exchange_before_agg()) { + return DataSinkOperatorX::required_data_distribution(state); + } return _is_colocate && _require_bucket_distribution ? DataDistribution(ExchangeType::BUCKET_HASH_SHUFFLE, _partition_exprs) : DataDistribution(ExchangeType::HASH_SHUFFLE, _partition_exprs); diff --git a/be/src/exec/operator/distinct_streaming_aggregation_operator.h b/be/src/exec/operator/distinct_streaming_aggregation_operator.h index 145ad4c79b0676..e0b175d93c551d 100644 --- a/be/src/exec/operator/distinct_streaming_aggregation_operator.h +++ b/be/src/exec/operator/distinct_streaming_aggregation_operator.h @@ -118,6 +118,10 @@ class DistinctStreamingAggOperatorX final if (_needs_finalize && _probe_expr_ctxs.empty()) { return {ExchangeType::NOOP}; } + if (!_needs_finalize && !state->enable_local_exchange_before_agg()) { + return StatefulOperatorX::required_data_distribution( + state); + } if (_needs_finalize || (!_probe_expr_ctxs.empty() && !_is_streaming_preagg)) { return _is_colocate && _require_bucket_distribution ? DataDistribution(ExchangeType::BUCKET_HASH_SHUFFLE, _partition_exprs) diff --git a/be/src/exec/operator/file_scan_operator.cpp b/be/src/exec/operator/file_scan_operator.cpp index abe89da95842b2..00e31e90691334 100644 --- a/be/src/exec/operator/file_scan_operator.cpp +++ b/be/src/exec/operator/file_scan_operator.cpp @@ -118,12 +118,9 @@ bool FileScanLocalState::_should_use_file_scanner_v2(const TQueryOptions& query_ const bool is_transactional_hive = scan_params.__isset.table_format_params && scan_params.table_format_params.table_format_type == "transactional_hive"; - // JNI reader selection is stored per split, but this scan-level selector cannot inspect the - // split yet. Older FEs may omit both the scan-level Paimon marker and split-level reader_type, - // so keep JNI scans on V1 until scanner selection can distinguish every compatibility shape. return query_options.__isset.enable_file_scanner_v2 && query_options.enable_file_scanner_v2 && - !is_load && scan_params.format_type != TFileFormatType::FORMAT_WAL && - scan_params.format_type != TFileFormatType::FORMAT_JNI && !is_transactional_hive; + !is_load && scan_params.format_type != TFileFormatType::FORMAT_ES_HTTP && + scan_params.format_type != TFileFormatType::FORMAT_LANCE && !is_transactional_hive; } Status FileScanLocalState::_init_scanners(std::list* scanners) { diff --git a/be/src/exec/operator/scan_operator.cpp b/be/src/exec/operator/scan_operator.cpp index f3b209ac2dd540..f945fa0a488810 100644 --- a/be/src/exec/operator/scan_operator.cpp +++ b/be/src/exec/operator/scan_operator.cpp @@ -73,34 +73,17 @@ bool ScanLocalState::should_run_serial() const { return _parent->cast()._should_run_serial; } -Status ScanLocalStateBase::update_late_arrival_runtime_filter( - RuntimeState* state, int applied_rf_num, int& arrived_rf_num, - VExprContextSPtrs& arrived_conjuncts) { +Status ScanLocalStateBase::update_late_arrival_runtime_filter(RuntimeState* state, + int& arrived_rf_num) { // Lock needed because _conjuncts can be accessed concurrently by multiple scanner threads LockGuard lock(_conjuncts_lock); - arrived_conjuncts.clear(); - size_t conjuncts_before = _conjuncts.size(); RETURN_IF_ERROR(_helper.try_append_late_arrival_runtime_filter(state, _parent->row_descriptor(), arrived_rf_num, _conjuncts)); - if (_conjuncts.size() > conjuncts_before) { - VExprContextSPtrs appended(_conjuncts.begin() + conjuncts_before, _conjuncts.end()); - _late_arrival_conjunct_batches.emplace_back(arrived_rf_num, std::move(appended)); - } if (state->enable_adjust_conjunct_order_by_cost()) { std::ranges::stable_sort(_conjuncts, [](const auto& a, const auto& b) { return a->execute_cost() < b->execute_cost(); }); }; - for (const auto& [batch_arrived_rf_num, batch] : _late_arrival_conjunct_batches) { - if (batch_arrived_rf_num <= applied_rf_num) { - continue; - } - for (const auto& conjunct : batch) { - VExprContextSPtr cloned; - RETURN_IF_ERROR(conjunct->clone(state, cloned)); - arrived_conjuncts.push_back(std::move(cloned)); - } - } return Status::OK(); } diff --git a/be/src/exec/operator/scan_operator.h b/be/src/exec/operator/scan_operator.h index a9a1f3f90f1dd9..ca9321644f108e 100644 --- a/be/src/exec/operator/scan_operator.h +++ b/be/src/exec/operator/scan_operator.h @@ -21,8 +21,6 @@ #include #include #include -#include -#include #include "common/status.h" #include "common/thread_safety_annotations.h" @@ -93,9 +91,7 @@ class ScanLocalStateBase : public PipelineXLocalState<> { uint64_t get_condition_cache_digest() const { return _condition_cache_digest; } - Status update_late_arrival_runtime_filter(RuntimeState* state, int applied_rf_num, - int& arrived_rf_num, - VExprContextSPtrs& arrived_conjuncts); + Status update_late_arrival_runtime_filter(RuntimeState* state, int& arrived_rf_num); Status clone_conjunct_ctxs(VExprContextSPtrs& scanner_conjuncts); @@ -134,9 +130,6 @@ class ScanLocalStateBase : public PipelineXLocalState<> { AnnotatedMutex _conjuncts_lock; RuntimeFilterConsumerHelper _helper; - // Preserve append identity independently of the cost-sorted operator snapshot. Every scanner - // needs the exact RF contexts added since its own applied count. - std::vector> _late_arrival_conjunct_batches; // magic number as seed to generate hash value for condition cache uint64_t _condition_cache_digest = 0; diff --git a/be/src/exec/operator/streaming_aggregation_operator.h b/be/src/exec/operator/streaming_aggregation_operator.h index 162b362145e805..5ed6d1481d5fcc 100644 --- a/be/src/exec/operator/streaming_aggregation_operator.h +++ b/be/src/exec/operator/streaming_aggregation_operator.h @@ -220,8 +220,7 @@ class StreamingAggOperatorX MOCK_REMOVE(final) : public StatefulOperatorXenable_streaming_agg_hash_join_force_passthrough()) { return DataDistribution(ExchangeType::PASSTHROUGH); } - if (!state->get_query_ctx()->should_be_shuffled_agg( - StatefulOperatorX::node_id())) { + if (!_needs_finalize && !state->enable_local_exchange_before_agg()) { return StatefulOperatorX::required_data_distribution(state); } if (_partition_exprs.empty()) { diff --git a/be/src/exec/scan/file_scanner_v2.cpp b/be/src/exec/scan/file_scanner_v2.cpp index a058aac64cef8f..c568832d67cddd 100644 --- a/be/src/exec/scan/file_scanner_v2.cpp +++ b/be/src/exec/scan/file_scanner_v2.cpp @@ -24,7 +24,6 @@ #include #include #include -#include #include #include @@ -61,6 +60,7 @@ #include "format_v2/table/paimon_reader.h" #include "format_v2/table/remote_doris_reader.h" #include "format_v2/table_reader.h" +#include "format_v2/wal/wal_table_reader.h" #include "io/cache/block_file_cache_profile.h" #include "io/fs/file_meta_cache.h" #include "io/io_common.h" @@ -108,10 +108,26 @@ bool is_supported_arrow_table_format(const TFileRangeDesc& range) { bool is_supported_jni_table_format(const TFileRangeDesc& range) { const auto table_format = table_format_name(range); if (table_format == "paimon") { - return range.__isset.table_format_params && - range.table_format_params.__isset.paimon_params && - range.table_format_params.paimon_params.__isset.reader_type && - range.table_format_params.paimon_params.reader_type == TPaimonReaderType::PAIMON_JNI; + if (!range.__isset.table_format_params || + !range.table_format_params.__isset.paimon_params) { + return false; + } + const auto& params = range.table_format_params.paimon_params; + if (params.__isset.reader_type) { + if (params.reader_type == TPaimonReaderType::PAIMON_JNI) { + return params.__isset.paimon_split; + } + // V2 cannot pass a logical DataSplit through a raw native child without silently + // dropping its multi-file semantics, so PAIMON_CPP must remain on the V1 fallback. + return false; + } + if (params.__isset.paimon_split) { + // Before reader_type was added, an encoded split unambiguously selected the Java + // reader; native scans carried only their physical Parquet or ORC range. + return true; + } + return params.__isset.file_format && + (params.file_format == "parquet" || params.file_format == "orc"); } return table_format == "jdbc" || table_format == "iceberg" || table_format == "hudi" || table_format == "max_compute" || table_format == "trino_connector"; @@ -155,6 +171,10 @@ bool is_native_format(TFileFormatType::type format_type) { return format_type == TFileFormatType::FORMAT_NATIVE; } +bool is_wal_format(TFileFormatType::type format_type) { + return format_type == TFileFormatType::FORMAT_WAL; +} + bool is_partition_slot(const TFileScanSlotInfo& slot_info, const std::string& column_name) { if (column_name.starts_with(BeConsts::GLOBAL_ROWID_COL) || column_name == BeConsts::ICEBERG_ROWID_COL) { @@ -233,9 +253,7 @@ Status adapt_runtime_filter_for_table_reader(VExprSPtr* expr) { #ifdef BE_TEST FileScannerV2::FileScannerV2(RuntimeState* state, RuntimeProfile* profile, std::unique_ptr table_reader) - : Scanner(state, profile), - _table_reader(std::move(table_reader)), - _scanner_profile(profile) {} + : Scanner(state, profile), _table_reader(std::move(table_reader)) {} Status FileScannerV2::TEST_validate_scan_range(const TFileScanRangeParams& params, const TFileRangeDesc& range) { @@ -302,6 +320,8 @@ bool FileScannerV2::is_supported(const TFileScanRangeParams& params, const TFile return is_supported_arrow_table_format(range); } else if (format_type == TFileFormatType::FORMAT_JNI) { return is_supported_jni_table_format(range); + } else if (is_wal_format(format_type)) { + return table_format_name(range) == "NotSet"; } else if (is_csv_format(format_type) || is_text_format(format_type) || is_json_format(format_type) || is_native_format(format_type)) { return is_supported_table_format(range); @@ -339,9 +359,7 @@ FileScannerV2::FileScannerV2(RuntimeState* state, FileScanLocalState* local_stat Status FileScannerV2::init(RuntimeState* state, const VExprContextSPtrs& conjuncts) { RETURN_IF_ERROR(Scanner::init(state, conjuncts)); - _initialize_scanner_residual_conjuncts(); auto* profile = _local_state->scanner_profile(); - _scanner_profile = profile; const auto hierarchy = file_scan_profile::ensure_hierarchy(profile); _scanner_total_timer = hierarchy.scanner; _io_timer = hierarchy.io; @@ -375,11 +393,6 @@ Status FileScannerV2::init(RuntimeState* state, const VExprContextSPtrs& conjunc profile, "AdaptiveBatchActualBytes", TUnit::BYTES, file_scan_profile::SCANNER, 1); _adaptive_batch_probe_count_counter = ADD_CHILD_COUNTER_WITH_LEVEL( profile, "AdaptiveBatchProbeCount", TUnit::UNIT, file_scan_profile::SCANNER, 1); - _scanner_residual_filter_timer = ADD_CHILD_TIMER_WITH_LEVEL( - profile, "ScannerResidualFilterTime", file_scan_profile::SCANNER, 1); - _scanner_residual_rows_filtered_counter = ADD_CHILD_COUNTER_WITH_LEVEL( - profile, "ScannerResidualRowsFiltered", TUnit::UNIT, file_scan_profile::SCANNER, 1); - _refresh_scanner_residual_profile(); SCOPED_TIMER(_scanner_total_timer); SCOPED_TIMER(_init_timer); _file_cache_statistics = std::make_unique(); @@ -421,7 +434,6 @@ Status FileScannerV2::_get_block_impl(RuntimeState* state, Block* block, bool* e SCOPED_TIMER(_get_block_timer); while (true) { RETURN_IF_CANCELLED(state); - RETURN_IF_ERROR(_sync_table_reader_conjuncts()); if (!_has_prepared_split) { RETURN_IF_ERROR(_prepare_next_split(eof)); if (*eof) { @@ -471,33 +483,18 @@ Status FileScannerV2::_get_block_impl(RuntimeState* state, Block* block, bool* e } Status FileScannerV2::_filter_output_block(Block* block) { - if (_scanner_residual_conjuncts.empty() || block->rows() == 0) { - return Status::OK(); - } - SCOPED_TIMER(_scanner_residual_filter_timer); - const size_t rows_before_filter = block->rows(); - auto status = VExprContext::filter_block(_scanner_residual_conjuncts, block, block->columns()); - if (!status.ok() && _params != nullptr && - _get_current_format_type() == TFileFormatType::FORMAT_ORC) { - status.prepend("Orc row reader nextBatch failed. reason = "); - } - RETURN_IF_ERROR(status); - const int64_t filtered_rows = cast_set(rows_before_filter - block->rows()); - _counter.num_rows_unselected += filtered_rows; - if (_scanner_residual_rows_filtered_counter != nullptr) { - COUNTER_UPDATE(_scanner_residual_rows_filtered_counter, filtered_rows); - } - return Status::OK(); -} - -size_t FileScannerV2::_last_block_rows_read(const Block& block) const { - const auto& stats = _table_reader->last_materialized_block_stats(); - return stats.has_materialized_input ? stats.rows : block.rows(); + return _contextualize_output_filter_status(Scanner::_filter_output_block(block), + _get_current_format_type()); } -size_t FileScannerV2::_last_block_bytes_read(const Block& block) const { - const auto& stats = _table_reader->last_materialized_block_stats(); - return stats.has_materialized_input ? stats.allocated_bytes : block.allocated_bytes(); +Status FileScannerV2::_contextualize_output_filter_status(Status status, + TFileFormatType::type format_type) { + if (!status.ok() && format_type == TFileFormatType::FORMAT_ORC) { + // Error-preserving expressions cannot be reordered into the ORC reader and therefore run + // at the scanner boundary; keep their error context identical to ORC callback failures. + status.prepend("Orc row reader nextBatch failed. reason = "); + } + return status; } Status FileScannerV2::_prepare_next_split(bool* eos) { @@ -583,7 +580,6 @@ Status FileScannerV2::_init_table_reader(const TFileRangeDesc& range) { RETURN_IF_ERROR(_table_reader->init({ .projected_columns = _projected_columns, .conjuncts = std::move(table_conjuncts), - .table_reader_owned_conjunct_count = _table_reader_owned_conjunct_count, .format = file_format, .scan_params = const_cast(_params), .io_ctx = _io_ctx, @@ -594,15 +590,17 @@ Status FileScannerV2::_init_table_reader(const TFileRangeDesc& range) { .push_down_count_columns = std::move(push_down_count_columns), .condition_cache_digest = _local_state->get_condition_cache_digest(), })); - _table_reader_applied_rf_num = _applied_rf_num; - // RFs collected before TableReader initialization are already present in the full snapshot. - _late_arrival_rf_conjuncts.clear(); return Status::OK(); } Status FileScannerV2::_create_table_reader_for_format( const TFileRangeDesc& range, std::unique_ptr* reader) const { DORIS_CHECK(reader != nullptr); + const auto file_format = get_range_format_type(*_params, range); + if (file_format == TFileFormatType::FORMAT_WAL) { + *reader = std::make_unique(); + return Status::OK(); + } const auto table_format = table_format_name(range); if (table_format == "NotSet" || table_format == "tvf") { *reader = std::make_unique(); @@ -641,12 +639,15 @@ Status FileScannerV2::_prepare_table_reader_split(const TFileRangeDesc& range, std::map partition_values) { format::FileFormat current_split_format; RETURN_IF_ERROR(_to_file_format(get_range_format_type(*_params, range), ¤t_split_format)); + VExprContextSPtrs conjuncts; + RETURN_IF_ERROR(_build_table_conjuncts(&conjuncts)); VExprContextSPtrs partition_prune_conjuncts; if (_state->query_options().enable_runtime_filter_partition_prune) { RETURN_IF_ERROR(_build_table_conjuncts(&partition_prune_conjuncts)); } RETURN_IF_ERROR(_table_reader->prepare_split({ .partition_values = std::move(partition_values), + .conjuncts = std::move(conjuncts), .partition_prune_conjuncts = std::move(partition_prune_conjuncts), // A metadata COUNT split may span scheduler turns. Do not enter that irreversible // synthetic-row path while a runtime filter can still arrive between batches. @@ -704,8 +705,8 @@ Status FileScannerV2::_generate_partition_values( DORIS_CHECK(range.columns_from_path_keys.size() == range.columns_from_path.size()); for (size_t idx = 0; idx < range.columns_from_path_keys.size(); ++idx) { const auto& key = range.columns_from_path_keys[idx]; - const auto it = _partition_slot_descs.find(key); - if (it == _partition_slot_descs.end()) { + const auto it = _split_value_slot_descs.find(key); + if (it == _split_value_slot_descs.end()) { continue; } const auto& value = range.columns_from_path[idx]; @@ -744,7 +745,7 @@ Status FileScannerV2::_parse_partition_value(const SlotDescriptor* slot_desc, Status FileScannerV2::_init_expr_ctxes() { _slot_id_to_desc.clear(); _slot_id_to_global_index.clear(); - _partition_slot_descs.clear(); + _split_value_slot_descs.clear(); _file_slot_descs.clear(); for (const auto* slot_desc : _output_tuple_desc->slots()) { _slot_id_to_desc.emplace(slot_desc->id(), slot_desc); @@ -775,6 +776,7 @@ Status FileScannerV2::_build_projected_columns(const format::TableReader& table_ slot_info.slot_id); } auto column = _build_table_column(it->second); + build_context.slot_desc = it->second; if (column.name.starts_with(BeConsts::GLOBAL_ROWID_COL)) { _need_global_rowid_column = true; } @@ -788,16 +790,16 @@ Status FileScannerV2::_build_projected_columns(const format::TableReader& table_ &column, it->second, build_context.schema_column.has_value() ? &*build_context.schema_column : nullptr, prefer_exact_name_match)); + _split_value_slot_descs.insert_or_assign( + column.name, + PartitionSlotInfo {.slot_desc = it->second, .canonical_name = column.name}); + for (const auto& alias : column.name_mapping) { + _split_value_slot_descs.emplace( + alias, + PartitionSlotInfo {.slot_desc = it->second, .canonical_name = column.name}); + } if (is_partition_slot(slot_info, column.name)) { column.is_partition_key = true; - _partition_slot_descs.emplace( - column.name, - PartitionSlotInfo {.slot_desc = it->second, .canonical_name = column.name}); - for (const auto& alias : column.name_mapping) { - _partition_slot_descs.emplace( - alias, - PartitionSlotInfo {.slot_desc = it->second, .canonical_name = column.name}); - } } else if (is_data_file_slot(slot_info, column.name)) { _file_slot_descs.push_back(const_cast(it->second)); } @@ -832,15 +834,10 @@ format::ColumnDefinition FileScannerV2::_build_table_column(const SlotDescriptor } Status FileScannerV2::_build_table_conjuncts(VExprContextSPtrs* conjuncts) const { - return _build_table_conjuncts(_conjuncts, conjuncts); -} - -Status FileScannerV2::_build_table_conjuncts(const VExprContextSPtrs& source, - VExprContextSPtrs* conjuncts) const { DORIS_CHECK(conjuncts != nullptr); conjuncts->clear(); - conjuncts->reserve(source.size()); - for (const auto& conjunct : source) { + conjuncts->reserve(_conjuncts.size()); + for (const auto& conjunct : _conjuncts) { VExprSPtr root; RETURN_IF_ERROR(format::clone_table_expr_tree(conjunct->root(), &root)); RETURN_IF_ERROR(rewrite_slot_refs_to_global_index(&root, _slot_id_to_global_index)); @@ -850,68 +847,6 @@ Status FileScannerV2::_build_table_conjuncts(const VExprContextSPtrs& source, return Status::OK(); } -size_t FileScannerV2::_safe_conjunct_prefix_size(const VExprContextSPtrs& conjuncts) { - for (size_t conjunct_index = 0; conjunct_index < conjuncts.size(); ++conjunct_index) { - if (!format::TableReader::is_safe_to_pre_execute(conjuncts[conjunct_index])) { - return conjunct_index; - } - } - return conjuncts.size(); -} - -void FileScannerV2::_initialize_scanner_residual_conjuncts() { - _table_reader_owned_conjunct_count = _safe_conjunct_prefix_size(_conjuncts); - // Preserve the entire suffix, not only the unsafe expression. Otherwise a later safe - // predicate could run below Scanner before a stateful/error-preserving ordering barrier. - _scanner_residual_conjuncts.assign( - _conjuncts.begin() + cast_set(_table_reader_owned_conjunct_count), - _conjuncts.end()); - _refresh_scanner_residual_profile(); -} - -void FileScannerV2::_refresh_scanner_residual_profile() { - if (_scanner_profile == nullptr || _scanner_residual_conjuncts.empty()) { - return; - } - std::ostringstream predicates; - predicates << "["; - for (size_t conjunct_index = 0; conjunct_index < _scanner_residual_conjuncts.size(); - ++conjunct_index) { - if (conjunct_index > 0) { - predicates << ", "; - } - predicates << _scanner_residual_conjuncts[conjunct_index]->root()->debug_string(); - } - predicates << "]"; - _scanner_profile->add_info_string("ScannerResidualPredicates", predicates.str()); -} - -Status FileScannerV2::_sync_table_reader_conjuncts() { - if (_table_reader == nullptr) { - return Status::OK(); - } - if (_table_reader_applied_rf_num == _applied_rf_num) { - return Status::OK(); - } - VExprContextSPtrs appended; - RETURN_IF_ERROR(_build_table_conjuncts(_late_arrival_rf_conjuncts, &appended)); - const size_t owned_count = _scanner_residual_conjuncts.empty() - ? _safe_conjunct_prefix_size(_late_arrival_rf_conjuncts) - : 0; - // Preserve existing expression state and append the identity-tracked RF delta. Cost sorting - // may move a late RF ahead of an old stateful predicate in the full scanner snapshot. - RETURN_IF_ERROR(_table_reader->append_conjuncts_with_ownership(appended, owned_count)); - _table_reader_owned_conjunct_count += owned_count; - _scanner_residual_conjuncts.insert( - _scanner_residual_conjuncts.end(), - _late_arrival_rf_conjuncts.begin() + cast_set(owned_count), - _late_arrival_rf_conjuncts.end()); - _refresh_scanner_residual_profile(); - _late_arrival_rf_conjuncts.clear(); - _table_reader_applied_rf_num = _applied_rf_num; - return Status::OK(); -} - TFileFormatType::type FileScannerV2::_get_current_format_type() const { return get_range_format_type(*_params, _current_range); } @@ -952,6 +887,9 @@ Status FileScannerV2::_to_file_format(TFileFormatType::type format_type, case TFileFormatType::FORMAT_ARROW: *file_format = format::FileFormat::ARROW; return Status::OK(); + case TFileFormatType::FORMAT_WAL: + *file_format = format::FileFormat::WAL; + return Status::OK(); default: return Status::NotSupported("FileScannerV2 does not support file format {}", to_string(format_type)); @@ -1036,19 +974,17 @@ void FileScannerV2::_update_adaptive_batch_size(const Block& block) { if (!_should_run_adaptive_batch_size()) { return; } - const auto& stats = _table_reader->last_materialized_block_stats(); - const size_t rows = stats.has_materialized_input ? stats.rows : block.rows(); - const size_t bytes = stats.has_materialized_input ? stats.bytes : block.bytes(); - COUNTER_SET(_adaptive_batch_actual_bytes_counter, static_cast(bytes)); - if (rows == 0) { + COUNTER_SET(_adaptive_batch_actual_bytes_counter, static_cast(block.bytes())); + if (block.rows() == 0) { return; } - // Residual predicates run after wide table columns are materialized. Learn from that pre-filter - // shape so selective predicates cannot make the next reader batch dangerously large. + // The sample is taken after TableReader has finalized file-local columns to table columns. + // This matches the memory shape seen by upstream operators and catches very wide nested + // columns, such as map/string payloads, after the first probe batch. if (!_block_size_predictor->has_history()) { COUNTER_UPDATE(_adaptive_batch_probe_count_counter, 1); } - _block_size_predictor->update(rows, bytes); + _block_size_predictor->update(block); } Status FileScannerV2::close(RuntimeState* state) { @@ -1240,8 +1176,9 @@ void FileScannerV2::_report_file_reader_predicate_filtered_rows() { const int64_t filtered_rows = _io_ctx != nullptr ? _io_ctx->predicate_filtered_rows : 0; const int64_t filtered_delta = filtered_rows - _reported_predicate_filtered_rows; if (filtered_delta > 0) { - // FileReader and TableReader both report their owned predicate rows through the shared IO - // context. Preserve scanner-level load statistics without re-evaluating either predicate. + // File readers can evaluate localized conjuncts before a block reaches Scanner. Count + // those rows as scanner-level unselected rows so load statistics stay identical no matter + // whether a predicate is pushed down or evaluated by Scanner::_filter_output_block(). _counter.num_rows_unselected += filtered_delta; _reported_predicate_filtered_rows = filtered_rows; } diff --git a/be/src/exec/scan/file_scanner_v2.h b/be/src/exec/scan/file_scanner_v2.h index cabbc96baa8400..658ec44ad7f051 100644 --- a/be/src/exec/scan/file_scanner_v2.h +++ b/be/src/exec/scan/file_scanner_v2.h @@ -89,22 +89,15 @@ class FileScannerV2 final : public Scanner { RuntimeProfile* profile, const io::FileCacheStatistics& file_cache_statistics); static bool TEST_should_skip_not_found(const Status& status, bool ignore_not_found); static bool TEST_should_skip_empty(const Status& status, bool stopped); + static Status TEST_contextualize_output_filter_status(Status status, + TFileFormatType::type format_type) { + return _contextualize_output_filter_status(std::move(status), format_type); + } static bool TEST_should_run_adaptive_batch_size(bool predictor_initialized, bool current_split_uses_metadata_count) { return _should_run_adaptive_batch_size(predictor_initialized, current_split_uses_metadata_count); } - void TEST_set_scanner_conjuncts(VExprContextSPtrs conjuncts) { - _conjuncts = std::move(conjuncts); - _initialize_scanner_residual_conjuncts(); - } - Status TEST_filter_output_block(Block* block) { return _filter_output_block(block); } - size_t TEST_table_reader_owned_conjunct_count() const { - return _table_reader_owned_conjunct_count; - } - size_t TEST_scanner_residual_conjunct_count() const { - return _scanner_residual_conjuncts.size(); - } #endif FileScannerV2(RuntimeState* state, FileScanLocalState* parent, int64_t limit, @@ -123,8 +116,6 @@ class FileScannerV2 final : public Scanner { protected: Status _get_block_impl(RuntimeState* state, Block* block, bool* eof) override; Status _filter_output_block(Block* block) override; - size_t _last_block_rows_read(const Block& block) const override; - size_t _last_block_bytes_read(const Block& block) const override; void _collect_profile_before_close() override; bool _should_update_load_counters() const override; @@ -143,6 +134,8 @@ class FileScannerV2 final : public Scanner { std::map partition_values); static bool _should_skip_not_found(const Status& status, bool ignore_not_found); static bool _should_skip_empty(const Status& status, bool stopped); + static Status _contextualize_output_filter_status(Status status, + TFileFormatType::type format_type); bool _should_enable_file_meta_cache() const; std::optional _create_global_rowid_context( const TFileRangeDesc& range) const; @@ -154,12 +147,6 @@ class FileScannerV2 final : public Scanner { Status _build_default_expr(const TFileScanSlotInfo& slot_info, VExprContextSPtr* ctx) const; static format::ColumnDefinition _build_table_column(const SlotDescriptor* slot_desc); Status _build_table_conjuncts(VExprContextSPtrs* conjuncts) const; - Status _build_table_conjuncts(const VExprContextSPtrs& source, - VExprContextSPtrs* conjuncts) const; - Status _sync_table_reader_conjuncts(); - static size_t _safe_conjunct_prefix_size(const VExprContextSPtrs& conjuncts); - void _initialize_scanner_residual_conjuncts(); - void _refresh_scanner_residual_profile(); static Status _to_file_format(TFileFormatType::type format_type, format::FileFormat* file_format); void _reset_adaptive_batch_size_state(); @@ -195,10 +182,6 @@ class FileScannerV2 final : public Scanner { std::string _current_range_path; std::unique_ptr _table_reader; - size_t _table_reader_owned_conjunct_count = 0; - // Scanner owns one persistent context vector for the first unsafe conjunct and every later - // conjunct. Hybrid child readers may be recreated or switched, but this state must not be. - VExprContextSPtrs _scanner_residual_conjuncts; std::vector _projected_columns; // File formats without embedded schema, such as CSV, still need the FE slot descriptors in // file-column order. This mirrors old FileScanner::_file_slot_descs and is passed only to @@ -207,7 +190,7 @@ class FileScannerV2 final : public Scanner { bool _need_global_rowid_column = false; std::unordered_map _slot_id_to_desc; std::unordered_map _slot_id_to_global_index; - std::unordered_map _partition_slot_descs; + std::unordered_map _split_value_slot_descs; std::unique_ptr _file_cache_statistics; io::FileCacheStatistics _reported_file_cache_statistics; @@ -232,9 +215,6 @@ class FileScannerV2 final : public Scanner { RuntimeProfile::Counter* _adaptive_batch_predicted_rows_counter = nullptr; RuntimeProfile::Counter* _adaptive_batch_actual_bytes_counter = nullptr; RuntimeProfile::Counter* _adaptive_batch_probe_count_counter = nullptr; - RuntimeProfile::Counter* _scanner_residual_filter_timer = nullptr; - RuntimeProfile::Counter* _scanner_residual_rows_filtered_counter = nullptr; - RuntimeProfile* _scanner_profile = nullptr; std::unique_ptr _block_size_predictor; int64_t _reported_predicate_filtered_rows = 0; int64_t _reported_condition_cache_hit_count = 0; @@ -244,7 +224,6 @@ class FileScannerV2 final : public Scanner { int64_t _last_bytes_read_from_local = 0; int64_t _last_bytes_read_from_remote = 0; int64_t _reported_io_read_time = 0; - int _table_reader_applied_rf_num = 0; }; } // namespace doris diff --git a/be/src/exec/scan/scanner.cpp b/be/src/exec/scan/scanner.cpp index c5a74b358e72da..3069438c764b90 100644 --- a/be/src/exec/scan/scanner.cpp +++ b/be/src/exec/scan/scanner.cpp @@ -19,8 +19,6 @@ #include -#include - #include "common/config.h" #include "common/status.h" #include "core/block/column_with_type_and_name.h" @@ -147,11 +145,8 @@ Status Scanner::get_block(RuntimeState* state, Block* block, bool* eof) { DCHECK(block->rows() == 0); break; } - // Some scanners apply owned predicates before returning the block. Account the - // materialized input, not only survivors, so the per-turn progress bound remains - // effective for highly selective predicates. - _num_rows_read += _last_block_rows_read(*block); - _num_byte_read += _last_block_bytes_read(*block); + _num_rows_read += block->rows(); + _num_byte_read += block->allocated_bytes(); } // 2. Filter the output block finally. @@ -233,9 +228,7 @@ Status Scanner::try_append_late_arrival_runtime_filter() { } DCHECK(_applied_rf_num < _total_rf_num); int arrived_rf_num = 0; - VExprContextSPtrs arrived_conjuncts; - RETURN_IF_ERROR(_local_state->update_late_arrival_runtime_filter( - _state, _applied_rf_num, arrived_rf_num, arrived_conjuncts)); + RETURN_IF_ERROR(_local_state->update_late_arrival_runtime_filter(_state, arrived_rf_num)); if (arrived_rf_num == _applied_rf_num) { // No newly arrived runtime filters, just return; @@ -245,9 +238,6 @@ Status Scanner::try_append_late_arrival_runtime_filter() { // avoid conjunct destroy in used by storage layer _conjuncts.clear(); RETURN_IF_ERROR(_local_state->clone_conjunct_ctxs(_conjuncts)); - _late_arrival_rf_conjuncts.insert(_late_arrival_rf_conjuncts.end(), - std::make_move_iterator(arrived_conjuncts.begin()), - std::make_move_iterator(arrived_conjuncts.end())); _applied_rf_num = arrived_rf_num; return Status::OK(); } diff --git a/be/src/exec/scan/scanner.h b/be/src/exec/scan/scanner.h index f12b6b2849f3ae..e90754db1c23d6 100644 --- a/be/src/exec/scan/scanner.h +++ b/be/src/exec/scan/scanner.h @@ -204,11 +204,6 @@ class Scanner { void update_block_avg_bytes(size_t block_avg_bytes) { _block_avg_bytes = block_avg_bytes; } protected: - virtual size_t _last_block_rows_read(const Block& block) const { return block.rows(); } - virtual size_t _last_block_bytes_read(const Block& block) const { - return block.allocated_bytes(); - } - RuntimeState* _state = nullptr; ScanLocalStateBase* _local_state = nullptr; @@ -236,9 +231,6 @@ class Scanner { // Cloned from _conjuncts of scan node. // It includes predicate in SQL and runtime filters. VExprContextSPtrs _conjuncts; - // Exact append-only RF delta for readers that preserve state across multiple splits. It must - // not be reconstructed by position from the cost-sorted full conjunct snapshot. - VExprContextSPtrs _late_arrival_rf_conjuncts; VExprContextSPtrs _projections; // Used in common subexpression elimination to compute intermediate results. std::vector _intermediate_projections; diff --git a/be/src/exprs/bloom_filter_func.h b/be/src/exprs/bloom_filter_func.h index d9e5ccab577afe..293680dcebe768 100644 --- a/be/src/exprs/bloom_filter_func.h +++ b/be/src/exprs/bloom_filter_func.h @@ -17,9 +17,12 @@ #pragma once +#include + #include "common/exception.h" #include "common/status.h" #include "core/column/column_dictionary.h" +#include "core/field.h" #include "exec/runtime_filter/runtime_filter_definitions.h" #include "exprs/bloom_filter_func_impl.h" #include "exprs/filter_base.h" @@ -137,6 +140,13 @@ class BloomFilterFuncBase : public FilterBase { virtual void find_fixed_len(const ColumnPtr& column, uint8_t* results, const uint8_t* __restrict filter = nullptr) = 0; + virtual PrimitiveType primitive_type() const = 0; + virtual bool supports_raw_fixed_values() const = 0; + virtual size_t raw_fixed_value_size() const = 0; + virtual Status find_batch_raw_fixed(const uint8_t* values, size_t rows, size_t value_width, + uint8_t* matches) const = 0; + virtual bool test_field(const Field& field) const = 0; + virtual uint16_t find_fixed_len_olap_engine(const IColumn& column, const uint8_t* nullmap, uint16_t* offsets, int number, bool is_parse_column) = 0; @@ -182,6 +192,56 @@ class BloomFilterFunc final : public BloomFilterFuncBase { OpV2::find_batch(*_bloom_filter, column, results, filter); } + PrimitiveType primitive_type() const override { return type; } + + bool supports_raw_fixed_values() const override { return !is_string_type(type); } + + size_t raw_fixed_value_size() const override { + if constexpr (is_string_type(type)) { + return 0; + } else { + return sizeof(typename PrimitiveTypeTraits::CppType); + } + } + + Status find_batch_raw_fixed(const uint8_t* values, size_t rows, size_t value_width, + uint8_t* matches) const override { + if constexpr (is_string_type(type)) { + return Status::NotSupported("String Bloom filter cannot probe fixed-width values"); + } else { + using ValueType = typename PrimitiveTypeTraits::CppType; + if (_bloom_filter == nullptr) { + return Status::InternalError("Bloom filter is not initialized"); + } + if (value_width != sizeof(ValueType)) { + return Status::Corruption("Raw Bloom filter width {} does not match expected {}", + value_width, sizeof(ValueType)); + } + DORIS_CHECK(values != nullptr || rows == 0); + DORIS_CHECK(matches != nullptr || rows == 0); + for (size_t row = 0; row < rows; ++row) { + ValueType value; + std::memcpy(&value, values + row * sizeof(ValueType), sizeof(ValueType)); + matches[row] &= _bloom_filter->test_element(value) ? 1 : 0; + } + return Status::OK(); + } + } + + bool test_field(const Field& field) const override { + DORIS_CHECK(_bloom_filter != nullptr); + if (field.is_null()) { + return _bloom_filter->contain_null(); + } + if constexpr (is_string_type(type)) { + const auto& value = field.get(); + return _bloom_filter->test_element( + StringRef(value.data(), value.size())); + } else { + return _bloom_filter->test_element(field.get()); + } + } + template uint16_t find_dict_olap_engine(const ColumnDictI32* column, const uint8_t* nullmap, uint16_t* offsets, int number) { diff --git a/be/src/exprs/function/functions_comparison.h b/be/src/exprs/function/functions_comparison.h index b5643e1ca5ebc5..1910bd845c791b 100644 --- a/be/src/exprs/function/functions_comparison.h +++ b/be/src/exprs/function/functions_comparison.h @@ -378,12 +378,47 @@ inline bool can_evaluate_equality(const VExprSPtrs& arguments, Op op) { return op == Op::EQ && can_evaluate(arguments); } +inline bool dictionary_value_matches(const Field& value, const Field& literal, Op op) { + switch (op) { + case Op::EQ: + return value == literal; + case Op::NE: + return value != literal; + case Op::LT: + return value < literal; + case Op::LE: + return value <= literal; + case Op::GT: + return value > literal; + case Op::GE: + return value >= literal; + } + __builtin_unreachable(); +} + inline ZoneMapFilterResult evaluate_dictionary(const DictionaryEvalContext& ctx, const VExprSPtrs& arguments, Op op) { - DORIS_CHECK(op == Op::EQ); auto slot_literal = expr_zonemap::extract_slot_and_literal(arguments); DORIS_CHECK(slot_literal.has_value()); - return expr_zonemap::eval_eq_dictionary(ctx, *slot_literal); + const auto* dictionary = ctx.slot(slot_literal->slot_index); + if (dictionary == nullptr || dictionary->data_type == nullptr) { + return ZoneMapFilterResult::kUnsupported; + } + DORIS_CHECK( + expr_zonemap::data_types_compatible(dictionary->data_type, slot_literal->slot_type)); + if (slot_literal->literal.is_null()) { + return ZoneMapFilterResult::kUnsupported; + } + const auto effective_op = slot_literal->literal_on_left ? symmetric_op(op) : op; + // Compare typed Fields so dictionary filtering preserves the regular expression semantics for + // strings, dates, decimals, floating-point edge cases, and every other supported logical type. + return std::ranges::any_of(dictionary->values, + [&](const Field& value) { + return dictionary_value_matches(value, slot_literal->literal, + effective_op); + }) + ? ZoneMapFilterResult::kMayMatch + : ZoneMapFilterResult::kNoMatch; } inline ZoneMapFilterResult evaluate_bloom_filter(const BloomFilterEvalContext& ctx, @@ -618,7 +653,7 @@ class FunctionComparison : public IFunction { bool can_evaluate_dictionary_filter(const VExprSPtrs& arguments) const override { auto op = comparison_zonemap_detail::op_from_name(name); - return op.has_value() && comparison_zonemap_detail::can_evaluate_equality(arguments, *op); + return op.has_value() && comparison_zonemap_detail::can_evaluate(arguments); } ZoneMapFilterResult evaluate_bloom_filter(const BloomFilterEvalContext& ctx, diff --git a/be/src/exprs/hybrid_set.h b/be/src/exprs/hybrid_set.h index bb4b2a50d6bb9e..33c0b57f0b31b2 100644 --- a/be/src/exprs/hybrid_set.h +++ b/be/src/exprs/hybrid_set.h @@ -20,6 +20,8 @@ #include #include +#include + #include "common/object_pool.h" #include "core/column/column_nullable.h" #include "core/column/column_string.h" @@ -198,6 +200,13 @@ class HybridSetBase : public FilterBase { // use in vectorize execute engine virtual bool find(const void* data, size_t) const = 0; + virtual void find_batch_raw_fixed(const uint8_t* values, size_t rows, size_t value_width, + uint8_t* matches) const { + for (size_t row = 0; row < rows; ++row) { + matches[row] &= find(values + row * value_width) ? 1 : 0; + } + } + virtual void find_batch(const doris::IColumn& column, size_t rows, doris::ColumnUInt8::Container& results, const uint8_t* __restrict filter = nullptr) = 0; @@ -291,6 +300,16 @@ class HybridSet : public HybridSetBase { bool find(const void* data, size_t /*unused*/) const override { return find(data); } + void find_batch_raw_fixed(const uint8_t* values, size_t rows, size_t value_width, + uint8_t* matches) const override { + DORIS_CHECK_EQ(value_width, sizeof(ElementType)); + for (size_t row = 0; row < rows; ++row) { + ElementType value; + std::memcpy(&value, values + row * sizeof(ElementType), sizeof(ElementType)); + matches[row] &= _set.find(value) ? 1 : 0; + } + } + void find_batch(const doris::IColumn& column, size_t rows, doris::ColumnUInt8::Container& results, const uint8_t* __restrict filter = nullptr) override { diff --git a/be/src/exprs/runtime_filter_expr.cpp b/be/src/exprs/runtime_filter_expr.cpp index a0d4403086af53..9df34915850e57 100644 --- a/be/src/exprs/runtime_filter_expr.cpp +++ b/be/src/exprs/runtime_filter_expr.cpp @@ -228,6 +228,37 @@ bool RuntimeFilterExpr::can_evaluate_zonemap_filter() const { return _impl->can_evaluate_zonemap_filter(); } +bool RuntimeFilterExpr::can_execute_on_raw_fixed_values(const DataTypePtr& data_type, + int column_id) const { + // Raw and dictionary streams omit NULL payloads and currently map NULL rows to false. A + // null-aware RF must therefore stay on execute_filter(), which restores its NULL semantics. + return !_null_aware && _impl->can_execute_on_raw_fixed_values(data_type, column_id); +} + +Status RuntimeFilterExpr::execute_on_raw_fixed_values(const uint8_t* values, size_t num_values, + size_t value_width, + const DataTypePtr& data_type, int column_id, + uint8_t* matches) const { + if (!can_execute_on_raw_fixed_values(data_type, column_id)) { + return Status::NotSupported("Runtime filter {} cannot evaluate raw fixed-width values", + _filter_id); + } + return _impl->execute_on_raw_fixed_values(values, num_values, value_width, data_type, column_id, + matches); +} + +ZoneMapFilterResult RuntimeFilterExpr::evaluate_dictionary_filter( + const DictionaryEvalContext& ctx) const { + if (!can_evaluate_dictionary_filter()) { + return ZoneMapFilterResult::kUnsupported; + } + return _impl->evaluate_dictionary_filter(ctx); +} + +bool RuntimeFilterExpr::can_evaluate_dictionary_filter() const { + return !_null_aware && _impl->can_evaluate_dictionary_filter(); +} + void RuntimeFilterExpr::collect_slot_column_ids(std::set& column_ids) const { _impl->collect_slot_column_ids(column_ids); } diff --git a/be/src/exprs/runtime_filter_expr.h b/be/src/exprs/runtime_filter_expr.h index 7994d2a71ae14f..4879b649a2f25d 100644 --- a/be/src/exprs/runtime_filter_expr.h +++ b/be/src/exprs/runtime_filter_expr.h @@ -107,6 +107,13 @@ class RuntimeFilterExpr final : public VExpr { ZoneMapFilterResult evaluate_zonemap_filter(const ZoneMapEvalContext& ctx) const override; bool can_evaluate_zonemap_filter() const override; + bool can_execute_on_raw_fixed_values(const DataTypePtr& data_type, + int column_id) const override; + Status execute_on_raw_fixed_values(const uint8_t* values, size_t num_values, size_t value_width, + const DataTypePtr& data_type, int column_id, + uint8_t* matches) const override; + ZoneMapFilterResult evaluate_dictionary_filter(const DictionaryEvalContext& ctx) const override; + bool can_evaluate_dictionary_filter() const override; void collect_slot_column_ids(std::set& column_ids) const override; int filter_id() const { return _filter_id; } diff --git a/be/src/exprs/vbloom_predicate.cpp b/be/src/exprs/vbloom_predicate.cpp index 4200371f918430..23501d2897ba87 100644 --- a/be/src/exprs/vbloom_predicate.cpp +++ b/be/src/exprs/vbloom_predicate.cpp @@ -31,6 +31,8 @@ #include "core/data_type/data_type_nullable.h" #include "core/types.h" #include "exprs/bloom_filter_func.h" +#include "exprs/expr_zonemap_filter.h" +#include "exprs/vslot_ref.h" #include "runtime/runtime_state.h" namespace doris { @@ -107,6 +109,71 @@ Status VBloomPredicate::execute_runtime_filter(VExprContext* context, const Bloc ColumnPtr* arg_column) const { return _do_execute(context, block, filter, nullptr, count, result_column); } + +namespace { + +bool bloom_filter_type_matches(PrimitiveType filter_type, const DataTypePtr& data_type) { + if (data_type == nullptr) { + return false; + } + const auto value_type = remove_nullable(data_type)->get_primitive_type(); + return filter_type == value_type || (is_string_type(filter_type) && is_string_type(value_type)); +} + +} // namespace + +bool VBloomPredicate::can_execute_on_raw_fixed_values(const DataTypePtr& data_type, + int column_id) const { + if (_filter == nullptr || !_filter->supports_raw_fixed_values() || _children.size() != 1) { + return false; + } + const auto slot = std::dynamic_pointer_cast(_children[0]); + return slot != nullptr && slot->column_id() == column_id && + bloom_filter_type_matches(_filter->primitive_type(), slot->data_type()) && + bloom_filter_type_matches(_filter->primitive_type(), data_type); +} + +Status VBloomPredicate::execute_on_raw_fixed_values(const uint8_t* values, size_t num_values, + size_t value_width, + const DataTypePtr& data_type, int column_id, + uint8_t* matches) const { + if (!can_execute_on_raw_fixed_values(data_type, column_id)) { + return Status::NotSupported("Bloom predicate cannot evaluate raw fixed-width values"); + } + // Hash physical values inside BloomFilterFunc; reconstructing an untyped hash here could + // disagree with the build-side hash for dates, decimals, and other fixed-width wrappers. + return _filter->find_batch_raw_fixed(values, num_values, value_width, matches); +} + +ZoneMapFilterResult VBloomPredicate::evaluate_dictionary_filter( + const DictionaryEvalContext& ctx) const { + if (!can_evaluate_dictionary_filter()) { + return ZoneMapFilterResult::kUnsupported; + } + const auto slot = std::dynamic_pointer_cast(_children[0]); + DORIS_CHECK(slot != nullptr); + const auto* dictionary = ctx.slot(slot->column_id()); + if (dictionary == nullptr || + !bloom_filter_type_matches(_filter->primitive_type(), dictionary->data_type)) { + return ZoneMapFilterResult::kUnsupported; + } + for (const auto& value : dictionary->values) { + if (_filter->test_field(value)) { + return ZoneMapFilterResult::kMayMatch; + } + } + return ZoneMapFilterResult::kNoMatch; +} + +bool VBloomPredicate::can_evaluate_dictionary_filter() const { + if (_filter == nullptr || _children.size() != 1) { + return false; + } + const auto slot = std::dynamic_pointer_cast(_children[0]); + return slot != nullptr && + bloom_filter_type_matches(_filter->primitive_type(), slot->data_type()); +} + const std::string& VBloomPredicate::expr_name() const { return EXPR_NAME; } @@ -127,4 +194,4 @@ uint64_t VBloomPredicate::get_digest(uint64_t seed) const { } #include "common/compile_check_end.h" -} // namespace doris \ No newline at end of file +} // namespace doris diff --git a/be/src/exprs/vbloom_predicate.h b/be/src/exprs/vbloom_predicate.h index f522fb4f29235b..9fca3ce7c7b50b 100644 --- a/be/src/exprs/vbloom_predicate.h +++ b/be/src/exprs/vbloom_predicate.h @@ -49,6 +49,14 @@ class VBloomPredicate final : public VExpr { const uint8_t* __restrict filter, size_t count, ColumnPtr& result_column, ColumnPtr* arg_column) const override; + bool can_execute_on_raw_fixed_values(const DataTypePtr& data_type, + int column_id) const override; + Status execute_on_raw_fixed_values(const uint8_t* values, size_t num_values, size_t value_width, + const DataTypePtr& data_type, int column_id, + uint8_t* matches) const override; + ZoneMapFilterResult evaluate_dictionary_filter(const DictionaryEvalContext& ctx) const override; + bool can_evaluate_dictionary_filter() const override; + Status prepare(RuntimeState* state, const RowDescriptor& desc, VExprContext* context) override; Status open(RuntimeState* state, VExprContext* context, FunctionContext::FunctionStateScope scope) override; diff --git a/be/src/exprs/vdirect_in_predicate.h b/be/src/exprs/vdirect_in_predicate.h index 8e47122e24f11a..7ff761c5aea484 100644 --- a/be/src/exprs/vdirect_in_predicate.h +++ b/be/src/exprs/vdirect_in_predicate.h @@ -99,6 +99,54 @@ class VDirectInPredicate final : public VExpr { std::dynamic_pointer_cast(get_child(0)) != nullptr; } + ZoneMapFilterResult evaluate_dictionary_filter( + const DictionaryEvalContext& ctx) const override { + return expr_zonemap::eval_in_dictionary(ctx, get_child(0), false, _seg_filter_values); + } + + bool can_evaluate_dictionary_filter() const override { + return _zonemap_materialized && + std::dynamic_pointer_cast(get_child(0)) != nullptr; + } + + bool can_execute_on_raw_fixed_values(const DataTypePtr& data_type, + int column_id) const override { + if (!_hybrid_set_values_match_child_type || data_type == nullptr || _filter == nullptr || + get_num_children() != 1) { + return false; + } + const auto slot = std::dynamic_pointer_cast(get_child(0)); + if (slot == nullptr || slot->column_id() != column_id) { + return false; + } + const auto raw_type = remove_nullable(data_type); + if (!remove_nullable(slot->data_type())->equals(*raw_type)) { + return false; + } + return _raw_fixed_value_size(raw_type->get_primitive_type()) != 0; + } + + Status execute_on_raw_fixed_values(const uint8_t* values, size_t num_values, size_t value_width, + const DataTypePtr& data_type, int column_id, + uint8_t* matches) const override { + if (!can_execute_on_raw_fixed_values(data_type, column_id)) { + return Status::NotSupported( + "Direct IN predicate cannot evaluate raw fixed-width values"); + } + DORIS_CHECK(values != nullptr || num_values == 0); + DORIS_CHECK(matches != nullptr || num_values == 0); + const size_t expected_width = + _raw_fixed_value_size(remove_nullable(data_type)->get_primitive_type()); + if (value_width != expected_width) { + return Status::Corruption("Raw direct IN width {} does not match expected {}", + value_width, expected_width); + } + // Dispatch once per decoder batch so large runtime-filter sets retain the typed HybridSet + // loop instead of paying a virtual lookup for every physical value. + _filter->find_batch_raw_fixed(values, num_values, value_width, matches); + return Status::OK(); + } + Status clone_node(VExprSPtr* cloned_expr) const override { DORIS_CHECK(cloned_expr != nullptr); *cloned_expr = VDirectInPredicate::create_shared(clone_texpr_node(), _filter, @@ -156,6 +204,37 @@ class VDirectInPredicate final : public VExpr { } private: + static size_t _raw_fixed_value_size(PrimitiveType primitive_type) { + switch (primitive_type) { +#define RETURN_RAW_FIXED_SIZE(TYPE) \ + case TYPE: \ + return sizeof(typename PrimitiveTypeTraits::CppType) + RETURN_RAW_FIXED_SIZE(TYPE_BOOLEAN); + RETURN_RAW_FIXED_SIZE(TYPE_TINYINT); + RETURN_RAW_FIXED_SIZE(TYPE_SMALLINT); + RETURN_RAW_FIXED_SIZE(TYPE_INT); + RETURN_RAW_FIXED_SIZE(TYPE_BIGINT); + RETURN_RAW_FIXED_SIZE(TYPE_LARGEINT); + RETURN_RAW_FIXED_SIZE(TYPE_FLOAT); + RETURN_RAW_FIXED_SIZE(TYPE_DOUBLE); + RETURN_RAW_FIXED_SIZE(TYPE_DATE); + RETURN_RAW_FIXED_SIZE(TYPE_DATETIME); + RETURN_RAW_FIXED_SIZE(TYPE_DATEV2); + RETURN_RAW_FIXED_SIZE(TYPE_DATETIMEV2); + RETURN_RAW_FIXED_SIZE(TYPE_TIMESTAMPTZ); + RETURN_RAW_FIXED_SIZE(TYPE_DECIMAL32); + RETURN_RAW_FIXED_SIZE(TYPE_DECIMAL64); + RETURN_RAW_FIXED_SIZE(TYPE_DECIMALV2); + RETURN_RAW_FIXED_SIZE(TYPE_DECIMAL128I); + RETURN_RAW_FIXED_SIZE(TYPE_DECIMAL256); + RETURN_RAW_FIXED_SIZE(TYPE_IPV4); + RETURN_RAW_FIXED_SIZE(TYPE_IPV6); +#undef RETURN_RAW_FIXED_SIZE + default: + return 0; + } + } + Status _do_execute(VExprContext* context, const Block* block, const uint8_t* __restrict filter, Selector* selector, size_t count, ColumnPtr& result_column, ColumnPtr* arg_column) const { diff --git a/be/src/exprs/vectorized_fn_call.cpp b/be/src/exprs/vectorized_fn_call.cpp index 58e3af4ad4c87b..ed7fa482145912 100644 --- a/be/src/exprs/vectorized_fn_call.cpp +++ b/be/src/exprs/vectorized_fn_call.cpp @@ -28,7 +28,6 @@ #include #include -#include "common/compare.h" #include "common/config.h" #include "common/exception.h" #include "common/logging.h" @@ -67,6 +66,7 @@ #include "storage/index/zone_map/zonemap_eval_context.h" #include "storage/segment/column_reader.h" #include "storage/segment/virtual_column_iterator.h" +#include "util/simd/parquet_kernels.h" namespace doris { class RowDescriptor; @@ -87,7 +87,7 @@ const static std::set OPS_FOR_ANN_RANGE_SEARCH = { namespace { -enum class RawComparisonOp : uint8_t { EQ, NE, LT, LE, GT, GE }; +using simd::RawComparisonOp; std::optional raw_comparison_op(std::string_view function_name, bool reverse) { RawComparisonOp op; @@ -129,34 +129,7 @@ template void execute_raw_comparison(const uint8_t* values, size_t num_values, const Field& literal, RawComparisonOp op, uint8_t* matches) { const T rhs = literal.get(); - for (size_t row = 0; row < num_values; ++row) { - if (matches[row] == 0) { - continue; - } - const T lhs = unaligned_load(values + row * sizeof(T)); - bool keep = false; - switch (op) { - case RawComparisonOp::EQ: - keep = Compare::equal(lhs, rhs); - break; - case RawComparisonOp::NE: - keep = Compare::not_equal(lhs, rhs); - break; - case RawComparisonOp::LT: - keep = Compare::less(lhs, rhs); - break; - case RawComparisonOp::LE: - keep = Compare::less_equal(lhs, rhs); - break; - case RawComparisonOp::GT: - keep = Compare::greater(lhs, rhs); - break; - case RawComparisonOp::GE: - keep = Compare::greater_equal(lhs, rhs); - break; - } - matches[row] = static_cast(keep); - } + simd::raw_compare(values, num_values, rhs, op, matches); } } // namespace diff --git a/be/src/format/table/iceberg_delete_file_reader_helper.cpp b/be/src/format/table/iceberg_delete_file_reader_helper.cpp index 7ee77e94eff255..8f0381fe768c05 100644 --- a/be/src/format/table/iceberg_delete_file_reader_helper.cpp +++ b/be/src/format/table/iceberg_delete_file_reader_helper.cpp @@ -174,6 +174,9 @@ Status init_orc_delete_reader(OrcReader* reader) { RETURN_IF_ERROR(reader->init_reader(&DELETE_COL_NAMES, &DELETE_COL_NAME_TO_BLOCK_IDX, conjuncts, false, nullptr, nullptr, nullptr, nullptr, TableSchemaChangeHelper::ConstNode::get_instance())); + // branch-4.1 creates the ORC row reader in set_fill_columns(); delete-file reads have no + // synthetic columns, but must still complete that initialization phase before nextBatch(). + RETURN_IF_ERROR(reader->set_fill_columns({}, {})); return Status::OK(); } diff --git a/be/src/format_v2/column_mapper.cpp b/be/src/format_v2/column_mapper.cpp index ed0dcda9739d6b..c3dbe3fa9e7766 100644 --- a/be/src/format_v2/column_mapper.cpp +++ b/be/src/format_v2/column_mapper.cpp @@ -399,33 +399,6 @@ std::string TableColumnMapperOptions::debug_string() const { return out.str(); } -bool requires_char_or_varchar_truncation(const ColumnMapping& mapping) { - if (mapping.table_type == nullptr) { - return false; - } - const auto table_type = remove_nullable(mapping.table_type); - const auto primitive_type = table_type->get_primitive_type(); - if (primitive_type != TYPE_VARCHAR && primitive_type != TYPE_CHAR) { - return false; - } - const auto target_len = assert_cast(table_type.get())->len(); - if (target_len <= 0) { - return false; - } - if (mapping.file_type == nullptr) { - return true; - } - const auto file_type = remove_nullable(mapping.file_type); - DORIS_CHECK(file_type != nullptr); - int file_len = -1; - if (file_type->get_primitive_type() == TYPE_VARCHAR || - file_type->get_primitive_type() == TYPE_CHAR || - file_type->get_primitive_type() == TYPE_STRING) { - file_len = assert_cast(file_type.get())->len(); - } - return file_len < 0 || target_len < file_len; -} - std::string ColumnDefinition::debug_string() const { std::ostringstream out; out << "ColumnDefinition{name=" << name << ", identifier=" << field_debug_string(identifier) @@ -2188,7 +2161,7 @@ Status TableColumnMapper::_build_filter_entries(const FileScanRequest& file_requ Status TableColumnMapper::create_scan_request( const std::vector& table_filters, const std::vector& projected_columns, FileScanRequest* file_request, - RuntimeState* runtime_state, FilterLocalizationResult* localization_result) { + RuntimeState* runtime_state) { // FileReader evaluates expressions against a file-local block. This mapper owns the // table-column to file-column conversion, so it also owns the file-local block positions. file_request->predicate_columns.clear(); @@ -2224,8 +2197,7 @@ Status TableColumnMapper::create_scan_request( // 2. Build referenced predicate columns // Hidden filter mappings must be built before localizing filters, so that they can be localized together with visible mappings and referenced by localized filter expressions. RETURN_IF_ERROR(_build_hidden_filter_mappings(table_filters)); - RETURN_IF_ERROR( - localize_filters(table_filters, file_request, runtime_state, localization_result)); + RETURN_IF_ERROR(localize_filters(table_filters, file_request, runtime_state)); for (const auto& mapping : _hidden_mappings) { if (!mapping.file_local_id.has_value()) { continue; @@ -2239,9 +2211,9 @@ Status TableColumnMapper::create_scan_request( if (is_visible_output) { continue; } - // A localized predicate is enforced exactly before TableReader materializes output. Only - // truly hidden mappings are absent from the final table block and may discard their - // payload after that file-local evaluation. + // File-local filtering is an optimization; Scanner still evaluates the original + // table-level conjunct after TableReader returns. Only truly hidden mappings are absent + // from that scanner-visible block and may safely discard their payload here. if (std::ranges::any_of(file_request->predicate_columns, [local_id](const LocalColumnIndex& projection) { return projection.column_id() == local_id; @@ -2290,11 +2262,7 @@ ColumnMapping* TableColumnMapper::_find_filter_mapping(GlobalIndex global_index) Status TableColumnMapper::localize_filters(const std::vector& table_filters, FileScanRequest* file_request, - RuntimeState* runtime_state, - FilterLocalizationResult* localization_result) { - if (localization_result != nullptr) { - localization_result->localized_filters.assign(table_filters.size(), false); - } + RuntimeState* runtime_state) { std::set localized_predicate_columns; FilterProjectionMap filter_projections; auto filter_mappings = _filter_visible_mappings(); @@ -2332,28 +2300,17 @@ Status TableColumnMapper::localize_filters(const std::vector& table // This keeps expression localization independent from filter iteration order. filter_mappings = _filter_visible_mappings(); const auto global_to_file_slot = build_file_slot_rewrite_map(filter_mappings, _filter_entries); - for (size_t filter_index = 0; filter_index < table_filters.size(); ++filter_index) { - const auto& table_filter = table_filters[filter_index]; + for (const auto& table_filter : table_filters) { if (table_filter.conjunct != nullptr && table_filter.conjunct->root() != nullptr) { const auto root = table_filter.conjunct->root(); const auto impl = root->get_impl(); const auto predicate = impl != nullptr ? impl : root; - if (!table_filter.can_localize || !predicate->is_deterministic() || + if (!predicate->is_deterministic() || !table_filter_has_only_local_entries(table_filter, _filter_entries)) { continue; } - if (runtime_state != nullptr && - runtime_state->query_options().truncate_char_or_varchar_columns && - std::ranges::any_of(table_filter.global_indices, [&](GlobalIndex global_index) { - const auto* mapping = _find_filter_mapping(global_index); - return mapping != nullptr && requires_char_or_varchar_truncation(*mapping); - })) { - // The table predicate observes the bounded value after finalize; evaluating it on - // a wider file string would change equality and range semantics. - continue; - } - // FileReader becomes the exact owner only for a stable predicate whose complete - // expression can be rewritten against this split's physical schema. + // Scanner evaluates the original conjunct after final materialization. Only predicates + // whose result is stable across repeated execution may also run as a file-local copy. RewriteContext rewrite_context {.runtime_state = runtime_state}; VExprSPtr rewrite_root; Status clone_status; @@ -2364,7 +2321,8 @@ Status TableColumnMapper::localize_filters(const std::vector& table // `element_at(MAP_VALUES(m)[1], 'age') > 30`. The current file-local rewrite only // understands top-level slots and struct-element paths rooted at top-level slots; // cloning such expressions can hit the generic TExpr complex-type limitation. - // Leave them for TableReader after final table-schema materialization. + // Leave them above TableReader, where Scanner evaluates the original table-level + // conjunct after final materialization. #ifndef NDEBUG return Status::InternalError( "Failed to clone table filter for file-local rewrite: {}, expr={}", @@ -2400,9 +2358,6 @@ Status TableColumnMapper::localize_filters(const std::vector& table auto localized_conjunct = VExprContext::create_shared(std::move(localized_root)); RETURN_IF_ERROR(rewrite_context.prepare_created_exprs(localized_conjunct.get())); file_request->conjuncts.push_back(std::move(localized_conjunct)); - if (localization_result != nullptr) { - localization_result->localized_filters[filter_index] = true; - } for (const auto global_index : table_filter.global_indices) { const auto* mapping = _find_filter_mapping(global_index); if (mapping != nullptr && mapping->file_local_id.has_value() && diff --git a/be/src/format_v2/column_mapper.h b/be/src/format_v2/column_mapper.h index 0860b687e4abc6..e03f836061ebf5 100644 --- a/be/src/format_v2/column_mapper.h +++ b/be/src/format_v2/column_mapper.h @@ -40,13 +40,6 @@ namespace doris::format { struct ColumnDefinition; struct TableFilter; -// Reports which table filters were fully rewritten into exact file-local predicates for the -// current split. The result is aligned with the TableFilter input vector and must not be reused for -// another split because schema evolution can change localization independently for every file. -struct FilterLocalizationResult { - std::vector localized_filters; -}; - enum class TableColumnMappingMode { // Match by ColumnDefinition::identifier TYPE_INT as field id. BY_FIELD_ID, @@ -172,8 +165,6 @@ struct TableColumnMapperOptions { std::string debug_string() const; }; -bool requires_char_or_varchar_truncation(const ColumnMapping& mapping); - Status clone_table_expr_tree(const VExprSPtr& expr, VExprSPtr* cloned_expr); const Field* find_partition_value(const ColumnDefinition& table_column, const std::map& partition_values); @@ -205,8 +196,7 @@ class TableColumnMapper { virtual Status create_scan_request(const std::vector& table_filters, const std::vector& projected_columns, FileScanRequest* file_request, - RuntimeState* runtime_state = nullptr, - FilterLocalizationResult* localization_result = nullptr); + RuntimeState* runtime_state = nullptr); // Localize table-level filters to the file schema. // Trivial mappings can copy structured predicates directly. Type changes may be localized with @@ -214,8 +204,7 @@ class TableColumnMapper { // table-level finalize/filter fallback. virtual Status localize_filters(const std::vector& table_filters, FileScanRequest* file_request, - RuntimeState* runtime_state = nullptr, - FilterLocalizationResult* localization_result = nullptr); + RuntimeState* runtime_state = nullptr); void clear() { _mappings.clear(); _hidden_mappings.clear(); diff --git a/be/src/format_v2/expr/cast.h b/be/src/format_v2/expr/cast.h index 1dc06bcf07f2bc..22604455e50099 100644 --- a/be/src/format_v2/expr/cast.h +++ b/be/src/format_v2/expr/cast.h @@ -53,6 +53,9 @@ class Cast final : public VExpr { std::string debug_string() const override; uint64_t get_digest(uint64_t seed) const override { return 0; } const std::string& expr_name() const override { return _expr_name; } + // Ordinary CAST can fail for data-dependent input. Localization must retain the same + // full-batch error behavior as VCastExpr instead of hiding errors in previously rejected rows. + bool is_safe_to_execute_on_selected_rows() const override { return false; } Status clone_node(VExprSPtr* cloned_expr) const override { DORIS_CHECK(cloned_expr != nullptr); *cloned_expr = Cast::create_shared(_data_type); diff --git a/be/src/format_v2/file_reader.h b/be/src/format_v2/file_reader.h index 1f90957064b254..3ff512975d2dcd 100644 --- a/be/src/format_v2/file_reader.h +++ b/be/src/format_v2/file_reader.h @@ -57,6 +57,7 @@ enum class FileFormat { JNI, NATIVE, ARROW, + WAL, }; struct FileScanRequest { @@ -76,9 +77,7 @@ struct FileScanRequest { std::vector predicate_only_columns; // file-local column id -> file-local output block position. std::map local_positions; - // Row-level filters converted to file-local expressions from table-level predicates. Readers - // must enforce these exactly on returned rows; metadata pruning alone does not transfer - // predicate ownership away from TableReader. + // Row-level filters converted to file-local expressions from table-level predicates. VExprContextSPtrs conjuncts; // Delete predicates converted to file-local expressions. A TRUE result means that the row is // deleted, so readers must invert each result when building their keep filter. diff --git a/be/src/format_v2/jni/jni_table_reader.cpp b/be/src/format_v2/jni/jni_table_reader.cpp index 7658a52fc86e0f..1c691fc3129c95 100644 --- a/be/src/format_v2/jni/jni_table_reader.cpp +++ b/be/src/format_v2/jni/jni_table_reader.cpp @@ -464,10 +464,16 @@ Status JniTableReader::_open_jni_scanner() { } void JniTableReader::set_batch_size(size_t batch_size) { - if (_scanner_opened && !supports_batch_size_update_after_open()) { - // Some connectors bake the constructor batch size into an already-open physical reader. - // Keep C++ and Java on that initial size instead of pretending a later resize took effect. - return; + if (!supports_batch_size_update_after_open()) { + if (_scanner_opened) { + return; + } + // Constructor-frozen readers must open with the stable query batch size; a transient + // adaptive probe would otherwise remain their physical batch size for the whole split. + if (_runtime_state != nullptr) { + TableReader::set_batch_size(_runtime_state->batch_size()); + return; + } } TableReader::set_batch_size(batch_size); if (!_scanner_opened) { diff --git a/be/src/format_v2/jni/paimon_jni_reader.cpp b/be/src/format_v2/jni/paimon_jni_reader.cpp index 47c0ef6c7bcac4..86d16ce6f7d7fd 100644 --- a/be/src/format_v2/jni/paimon_jni_reader.cpp +++ b/be/src/format_v2/jni/paimon_jni_reader.cpp @@ -59,7 +59,7 @@ Status PaimonJniReader::validate_scan_range(const TFileRangeDesc& range) const { "missing paimon_split for paimon jni reader, possibly caused by FE/BE protocol " "mismatch"); } - if (!range.table_format_params.paimon_params.__isset.reader_type || + if (range.table_format_params.paimon_params.__isset.reader_type && range.table_format_params.paimon_params.reader_type != TPaimonReaderType::PAIMON_JNI) { return Status::InternalError( "invalid reader_type for paimon jni reader, possibly caused by FE/BE protocol " diff --git a/be/src/format_v2/parquet/parquet_profile.cpp b/be/src/format_v2/parquet/parquet_profile.cpp index b56a57d419e0f3..6bf354f63bb2fc 100644 --- a/be/src/format_v2/parquet/parquet_profile.cpp +++ b/be/src/format_v2/parquet/parquet_profile.cpp @@ -175,10 +175,20 @@ void ParquetProfile::init(RuntimeProfile* profile) { TUnit::BYTES, parquet_profile, 1); predicate_compaction_count = ADD_CHILD_COUNTER_WITH_LEVEL(profile, "PredicateCompactionCount", TUnit::UNIT, parquet_profile, 1); + predicate_alignment_columns = ADD_CHILD_COUNTER_WITH_LEVEL(profile, "PredicateAlignmentColumns", + TUnit::UNIT, parquet_profile, 1); fixed_width_predicate_direct_batches = ADD_CHILD_COUNTER_WITH_LEVEL( profile, "FixedWidthPredicateDirectBatches", TUnit::UNIT, parquet_profile, 1); fixed_width_predicate_direct_rows = ADD_CHILD_COUNTER_WITH_LEVEL( profile, "FixedWidthPredicateDirectRows", TUnit::UNIT, parquet_profile, 1); + dictionary_predicate_direct_batches = ADD_CHILD_COUNTER_WITH_LEVEL( + profile, "DictionaryPredicateDirectBatches", TUnit::UNIT, parquet_profile, 1); + dictionary_predicate_direct_rows = ADD_CHILD_COUNTER_WITH_LEVEL( + profile, "DictionaryPredicateDirectRows", TUnit::UNIT, parquet_profile, 1); + dictionary_predicate_projected_rows = ADD_CHILD_COUNTER_WITH_LEVEL( + profile, "DictionaryPredicateProjectedRows", TUnit::UNIT, parquet_profile, 1); + dictionary_predicate_fused_projected_rows = ADD_CHILD_COUNTER_WITH_LEVEL( + profile, "DictionaryPredicateFusedProjectedRows", TUnit::UNIT, parquet_profile, 1); dict_filter_rewrite_time = ADD_CHILD_TIMER_WITH_LEVEL(profile, "DictFilterRewriteTime", parquet_profile, 1); dict_filter_expr_rewrite_time = @@ -191,6 +201,10 @@ void ParquetProfile::init(RuntimeProfile* profile) { profile, "DictFilterCandidateColumns", TUnit::UNIT, parquet_profile, 1); dict_filter_columns = ADD_CHILD_COUNTER_WITH_LEVEL(profile, "DictFilterColumns", TUnit::UNIT, parquet_profile, 1); + dict_filter_typed_compare_columns = ADD_CHILD_COUNTER_WITH_LEVEL( + profile, "DictFilterTypedCompareColumns", TUnit::UNIT, parquet_profile, 1); + dict_filter_string_compare_columns = ADD_CHILD_COUNTER_WITH_LEVEL( + profile, "DictFilterStringCompareColumns", TUnit::UNIT, parquet_profile, 1); dict_filter_unsupported_columns = ADD_CHILD_COUNTER_WITH_LEVEL( profile, "DictFilterUnsupportedColumns", TUnit::UNIT, parquet_profile, 1); dict_filter_read_failures = ADD_CHILD_COUNTER_WITH_LEVEL(profile, "DictFilterReadFailures", @@ -274,6 +288,7 @@ ParquetColumnReaderProfile ParquetProfile::column_reader_profile() const { .hybrid_selection_batches = hybrid_selection_batches, .hybrid_selection_ranges = hybrid_selection_ranges, .hybrid_selection_null_fallback_batches = hybrid_selection_null_fallback_batches, + .dictionary_predicate_fused_projected_rows = dictionary_predicate_fused_projected_rows, .decompress_time = decompress_time, .decompress_count = decompress_cnt, .decode_header_time = decode_header_time, @@ -316,14 +331,20 @@ ParquetScanProfile ParquetProfile::scan_profile() const { .predicate_compaction_time = predicate_compaction_time, .predicate_compaction_bytes = predicate_compaction_bytes, .predicate_compaction_count = predicate_compaction_count, + .predicate_alignment_columns = predicate_alignment_columns, .fixed_width_predicate_direct_batches = fixed_width_predicate_direct_batches, .fixed_width_predicate_direct_rows = fixed_width_predicate_direct_rows, + .dictionary_predicate_direct_batches = dictionary_predicate_direct_batches, + .dictionary_predicate_direct_rows = dictionary_predicate_direct_rows, + .dictionary_predicate_projected_rows = dictionary_predicate_projected_rows, .dict_filter_rewrite_time = dict_filter_rewrite_time, .dict_filter_expr_rewrite_time = dict_filter_expr_rewrite_time, .dict_filter_read_dict_time = dict_filter_read_dict_time, .dict_filter_build_time = dict_filter_build_time, .dict_filter_candidate_columns = dict_filter_candidate_columns, .dict_filter_columns = dict_filter_columns, + .dict_filter_typed_compare_columns = dict_filter_typed_compare_columns, + .dict_filter_string_compare_columns = dict_filter_string_compare_columns, .dict_filter_unsupported_columns = dict_filter_unsupported_columns, .dict_filter_read_failures = dict_filter_read_failures, .rows_filtered_by_dict_filter = rows_filtered_by_dict_filter, diff --git a/be/src/format_v2/parquet/parquet_profile.h b/be/src/format_v2/parquet/parquet_profile.h index 170f14b56c5d14..80198bc7a148a4 100644 --- a/be/src/format_v2/parquet/parquet_profile.h +++ b/be/src/format_v2/parquet/parquet_profile.h @@ -41,6 +41,7 @@ struct ParquetColumnReaderProfile { RuntimeProfile::Counter* hybrid_selection_batches = nullptr; RuntimeProfile::Counter* hybrid_selection_ranges = nullptr; RuntimeProfile::Counter* hybrid_selection_null_fallback_batches = nullptr; + RuntimeProfile::Counter* dictionary_predicate_fused_projected_rows = nullptr; // Native page/encoding reader internals. These counters keep page IO, decompression, levels, // value decode and conversion attributable to separate stages. RuntimeProfile::Counter* decompress_time = nullptr; @@ -88,16 +89,24 @@ struct ParquetScanProfile { RuntimeProfile::Counter* predicate_compaction_time = nullptr; RuntimeProfile::Counter* predicate_compaction_bytes = nullptr; RuntimeProfile::Counter* predicate_compaction_count = nullptr; + RuntimeProfile::Counter* predicate_alignment_columns = nullptr; RuntimeProfile::Counter* fixed_width_predicate_direct_batches = nullptr; RuntimeProfile::Counter* fixed_width_predicate_direct_rows = nullptr; + RuntimeProfile::Counter* dictionary_predicate_direct_batches = nullptr; + RuntimeProfile::Counter* dictionary_predicate_direct_rows = nullptr; + RuntimeProfile::Counter* dictionary_predicate_projected_rows = nullptr; RuntimeProfile::Counter* dict_filter_rewrite_time = nullptr; // dictionary rewrite time (ns) RuntimeProfile::Counter* dict_filter_expr_rewrite_time = nullptr; // expression/residual rewrite time (ns) RuntimeProfile::Counter* dict_filter_read_dict_time = nullptr; // dictionary page read time (ns) RuntimeProfile::Counter* dict_filter_build_time = nullptr; // dictionary entry bitmap build time (ns) - RuntimeProfile::Counter* dict_filter_candidate_columns = nullptr; // candidate columns - RuntimeProfile::Counter* dict_filter_columns = nullptr; // optimized columns + RuntimeProfile::Counter* dict_filter_candidate_columns = nullptr; // candidate columns + RuntimeProfile::Counter* dict_filter_columns = nullptr; // optimized columns + RuntimeProfile::Counter* dict_filter_typed_compare_columns = + nullptr; // fixed-width typed comparison columns + RuntimeProfile::Counter* dict_filter_string_compare_columns = + nullptr; // string typed comparison columns RuntimeProfile::Counter* dict_filter_unsupported_columns = nullptr; // unsupported columns RuntimeProfile::Counter* dict_filter_read_failures = nullptr; // dictionary read failures RuntimeProfile::Counter* rows_filtered_by_dict_filter = nullptr; // rows filtered by dict @@ -153,6 +162,7 @@ struct ParquetProfile { RuntimeProfile::Counter* hybrid_selection_batches = nullptr; RuntimeProfile::Counter* hybrid_selection_ranges = nullptr; RuntimeProfile::Counter* hybrid_selection_null_fallback_batches = nullptr; + RuntimeProfile::Counter* dictionary_predicate_fused_projected_rows = nullptr; RuntimeProfile::Counter* native_read_calls = nullptr; RuntimeProfile::Counter* native_page_fragments = nullptr; RuntimeProfile::Counter* page_crossing_batches = nullptr; @@ -203,14 +213,20 @@ struct ParquetProfile { RuntimeProfile::Counter* predicate_compaction_time = nullptr; RuntimeProfile::Counter* predicate_compaction_bytes = nullptr; RuntimeProfile::Counter* predicate_compaction_count = nullptr; + RuntimeProfile::Counter* predicate_alignment_columns = nullptr; RuntimeProfile::Counter* fixed_width_predicate_direct_batches = nullptr; RuntimeProfile::Counter* fixed_width_predicate_direct_rows = nullptr; + RuntimeProfile::Counter* dictionary_predicate_direct_batches = nullptr; + RuntimeProfile::Counter* dictionary_predicate_direct_rows = nullptr; + RuntimeProfile::Counter* dictionary_predicate_projected_rows = nullptr; RuntimeProfile::Counter* dict_filter_rewrite_time = nullptr; RuntimeProfile::Counter* dict_filter_expr_rewrite_time = nullptr; RuntimeProfile::Counter* dict_filter_read_dict_time = nullptr; RuntimeProfile::Counter* dict_filter_build_time = nullptr; RuntimeProfile::Counter* dict_filter_candidate_columns = nullptr; RuntimeProfile::Counter* dict_filter_columns = nullptr; + RuntimeProfile::Counter* dict_filter_typed_compare_columns = nullptr; + RuntimeProfile::Counter* dict_filter_string_compare_columns = nullptr; RuntimeProfile::Counter* dict_filter_unsupported_columns = nullptr; RuntimeProfile::Counter* dict_filter_read_failures = nullptr; RuntimeProfile::Counter* rows_filtered_by_dict_filter = nullptr; diff --git a/be/src/format_v2/parquet/parquet_scan.cpp b/be/src/format_v2/parquet/parquet_scan.cpp index b92a3c27054ad1..667cc4de41f9dd 100644 --- a/be/src/format_v2/parquet/parquet_scan.cpp +++ b/be/src/format_v2/parquet/parquet_scan.cpp @@ -22,6 +22,7 @@ #include #include #include +#include #include #include @@ -29,8 +30,11 @@ #include "common/status.h" #include "core/assert_cast.h" #include "core/block/block.h" +#include "core/column/column_decimal.h" #include "core/column/column_vector.h" +#include "exprs/expr_zonemap_filter.h" #include "exprs/vcompound_pred.h" +#include "exprs/vectorized_fn_call.h" #include "exprs/vexpr_context.h" #include "format_v2/parquet/parquet_column_schema.h" #include "format_v2/parquet/parquet_file_context.h" @@ -569,8 +573,8 @@ Status finalize_parquet_row_group_plans( namespace { -using DictionaryResidualConjunct = std::pair; -using DictionaryResidualConjuncts = std::vector; +using OwnedExpressionConjunct = std::pair; +using OwnedExpressionConjuncts = std::vector; void update_counter_if_not_null(RuntimeProfile::Counter* counter, int64_t value) { if (counter != nullptr) { @@ -615,10 +619,9 @@ Status execute_compact_filter_conjuncts(const VExprContextSPtrs& conjuncts, size return Status::OK(); } -Status execute_compact_dictionary_residual_conjuncts(const DictionaryResidualConjuncts& conjuncts, - size_t rows, Block* file_block, - IColumn::Filter* compact_filter, - bool* can_filter_all) { +Status execute_compact_owned_conjuncts(std::span conjuncts, + size_t rows, Block* file_block, + IColumn::Filter* compact_filter, bool* can_filter_all) { DORIS_CHECK(compact_filter != nullptr); DORIS_CHECK(can_filter_all != nullptr); compact_filter->resize_fill(rows, 1); @@ -794,11 +797,17 @@ int64_t count_range_rows(const std::vector& ranges) { } void append_intersection(const RowRange& left, const RowRange& right, - std::vector* result) { + std::vector& result) { const int64_t start = std::max(left.start, right.start); const int64_t end = std::min(left.start + left.length, right.start + right.length); if (start < end) { - result->push_back(RowRange {.start = start, .length = end - start}); + // Cache granules are only filter coordinates. Merge adjacent survivors so cache hits + // preserve the original read-range batch boundaries, matching V1 RowRanges semantics. + if (!result.empty() && result.back().start + result.back().length == start) { + result.back().length = end - result.back().start; + return; + } + result.push_back(RowRange {.start = start, .length = end - start}); } } @@ -830,7 +839,7 @@ std::vector filter_ranges_by_condition_cache(const std::vector ParquetScanScheduler::adaptive_predicate_prefetch_columns( - const format::FileScanRequest& request) const { + const format::FileScanRequest& request) { std::vector positions; std::unordered_map columns_by_position; - positions.reserve(request.predicate_columns.size()); columns_by_position.reserve(request.predicate_columns.size()); for (const auto& column : request.predicate_columns) { const auto position_it = request.local_positions.find(column.column_id()); DORIS_CHECK(position_it != request.local_positions.end()); const size_t position = position_it->second.value(); - positions.push_back(position); columns_by_position.emplace(position, &column); } + const auto& schedule = predicate_conjunct_schedule(request); + if (!schedule.supports_lazy_materialization) { + positions.reserve(request.predicate_columns.size()); + for (const auto& column : request.predicate_columns) { + positions.push_back(request.local_positions.at(column.column_id()).value()); + } + } else if (!schedule.single_column_conjuncts.empty()) { + positions.reserve(schedule.single_column_conjuncts.size()); + for (const auto& column : request.predicate_columns) { + const size_t position = request.local_positions.at(column.column_id()).value(); + if (schedule.single_column_conjuncts.contains(position)) { + // Cold adaptive statistics intentionally preserve request order; iterating the + // hash map here would make the first decoded predicate depend on bucket layout. + positions.push_back(position); + } + } + } else if (!schedule.remaining_stages.empty()) { + // Match execution's first reachable stage. Warming columns owned only by later residuals + // would turn lazy decode into eager remote IO before an earlier conjunct can reject rows. + positions = schedule.remaining_stages.front().required_positions; + } else { + positions.reserve(request.predicate_columns.size()); + for (const auto& column : request.predicate_columns) { + positions.push_back(request.local_positions.at(column.column_id()).value()); + } + } auto ordered = detail::order_adaptive_predicates(positions, _predicate_runtime_stats); ordered = detail::adaptive_prefetch_prefix(ordered, _predicate_runtime_stats, 0.25); std::vector result; @@ -1214,6 +1250,37 @@ Status ParquetScanScheduler::flush_pending_non_predicate_skip_rows() { namespace { +bool append_residual_stages(const VExprContextSPtr& owner_context, const VExprSPtr& expression, + const std::unordered_set& predicate_block_positions, + std::vector* stages) { + DORIS_CHECK(owner_context != nullptr); + DORIS_CHECK(expression != nullptr); + DORIS_CHECK(stages != nullptr); + const auto* compound_predicate = dynamic_cast(expression.get()); + if (compound_predicate != nullptr && compound_predicate->op() == TExprOpcode::COMPOUND_AND) { + for (const auto& child : expression->children()) { + if (!append_residual_stages(owner_context, child, predicate_block_positions, stages)) { + return false; + } + } + return true; + } + + std::set referenced_positions; + expression->collect_slot_column_ids(referenced_positions); + auto& stage = stages->emplace_back(); + stage.owner_context = owner_context; + stage.expression = expression; + for (const int position : referenced_positions) { + if (position < 0 || !predicate_block_positions.contains(cast_set(position))) { + stages->pop_back(); + return false; + } + stage.required_positions.push_back(cast_set(position)); + } + return true; +} + detail::PredicateConjunctSchedule build_predicate_conjunct_schedule( const format::FileScanRequest& request) { std::unordered_set predicate_block_positions; @@ -1235,18 +1302,31 @@ detail::PredicateConjunctSchedule build_predicate_conjunct_schedule( // optimization, so any unsafe conjunct disables the per-column schedule for the batch. schedule.remaining_conjuncts = request.conjuncts; schedule.single_column_conjuncts.clear(); + schedule.remaining_stages.clear(); + schedule.supports_lazy_materialization = false; return schedule; } std::set referenced_positions; conjunct->root()->collect_slot_column_ids(referenced_positions); if (referenced_positions.size() != 1) { schedule.remaining_conjuncts.push_back(conjunct); + if (!append_residual_stages(conjunct, conjunct->root(), predicate_block_positions, + &schedule.remaining_stages)) { + schedule.supports_lazy_materialization = false; + schedule.remaining_conjuncts = request.conjuncts; + schedule.single_column_conjuncts.clear(); + schedule.remaining_stages.clear(); + return schedule; + } continue; } const auto block_position = static_cast(*referenced_positions.begin()); if (!predicate_block_positions.contains(block_position)) { - schedule.remaining_conjuncts.push_back(conjunct); - continue; + schedule.supports_lazy_materialization = false; + schedule.remaining_conjuncts = request.conjuncts; + schedule.single_column_conjuncts.clear(); + schedule.remaining_stages.clear(); + return schedule; } schedule.single_column_conjuncts[block_position].push_back(conjunct); } @@ -1280,7 +1360,7 @@ bool can_evaluate_dictionary_exactly(const VExprSPtr& expr) { } void collect_dictionary_residual_exprs(const VExprContextSPtr& owner_context, const VExprSPtr& expr, - DictionaryResidualConjuncts* residual_conjuncts) { + OwnedExpressionConjuncts* residual_conjuncts) { DORIS_CHECK(owner_context != nullptr); DORIS_CHECK(expr != nullptr); DORIS_CHECK(residual_conjuncts != nullptr); @@ -1304,9 +1384,8 @@ void collect_dictionary_residual_exprs(const VExprContextSPtr& owner_context, co residual_conjuncts->emplace_back(owner_context, expr); } -DictionaryResidualConjuncts build_dictionary_residual_conjuncts( - const VExprContextSPtrs& conjuncts) { - DictionaryResidualConjuncts residual_conjuncts; +OwnedExpressionConjuncts build_dictionary_residual_conjuncts(const VExprContextSPtrs& conjuncts) { + OwnedExpressionConjuncts residual_conjuncts; for (const auto& conjunct : conjuncts) { DORIS_CHECK(conjunct != nullptr); collect_dictionary_residual_exprs(conjunct, conjunct->root(), &residual_conjuncts); @@ -1322,14 +1401,195 @@ uint16_t count_selected_rows(const IColumn::Filter& filter) { return selected_rows; } -IColumn::Filter build_dictionary_entry_filter(size_t block_position, - const ParquetColumnSchema& column_schema, - const VExprContextSPtrs& conjuncts, - const IColumn& dictionary) { - IColumn::Filter dictionary_filter(dictionary.size(), 1); +enum class DictionaryEntryFilterKernel { + GENERIC, + TYPED_FIXED_WIDTH, + TYPED_STRING, +}; + +template +bool get_fixed_dictionary_raw_values(const IColumn& dictionary, const uint8_t** values, + size_t* value_width) { + const auto* typed_dictionary = check_and_get_column(dictionary); + if (typed_dictionary == nullptr) { + return false; + } + *values = reinterpret_cast(typed_dictionary->get_data().data()); + *value_width = sizeof(typename ColumnType::value_type); + return true; +} + +bool get_typed_dictionary_raw_values(PrimitiveType primitive_type, const IColumn& dictionary, + const uint8_t** values, size_t* value_width) { + switch (primitive_type) { +#define GET_TYPED_DICTIONARY_VALUES(TYPE) \ + case TYPE: \ + return get_fixed_dictionary_raw_values::ColumnType>( \ + dictionary, values, value_width) + GET_TYPED_DICTIONARY_VALUES(TYPE_BOOLEAN); + GET_TYPED_DICTIONARY_VALUES(TYPE_TINYINT); + GET_TYPED_DICTIONARY_VALUES(TYPE_SMALLINT); + GET_TYPED_DICTIONARY_VALUES(TYPE_INT); + GET_TYPED_DICTIONARY_VALUES(TYPE_BIGINT); + GET_TYPED_DICTIONARY_VALUES(TYPE_LARGEINT); + GET_TYPED_DICTIONARY_VALUES(TYPE_FLOAT); + GET_TYPED_DICTIONARY_VALUES(TYPE_DOUBLE); + GET_TYPED_DICTIONARY_VALUES(TYPE_DATE); + GET_TYPED_DICTIONARY_VALUES(TYPE_DATETIME); + GET_TYPED_DICTIONARY_VALUES(TYPE_DATEV2); + GET_TYPED_DICTIONARY_VALUES(TYPE_DATETIMEV2); + GET_TYPED_DICTIONARY_VALUES(TYPE_TIMESTAMPTZ); + GET_TYPED_DICTIONARY_VALUES(TYPE_DECIMAL32); + GET_TYPED_DICTIONARY_VALUES(TYPE_DECIMAL64); + GET_TYPED_DICTIONARY_VALUES(TYPE_DECIMALV2); + GET_TYPED_DICTIONARY_VALUES(TYPE_DECIMAL128I); + GET_TYPED_DICTIONARY_VALUES(TYPE_DECIMAL256); + GET_TYPED_DICTIONARY_VALUES(TYPE_IPV4); + GET_TYPED_DICTIONARY_VALUES(TYPE_IPV6); +#undef GET_TYPED_DICTIONARY_VALUES + default: + return false; + } +} + +enum class StringDictionaryCompareOp { + EQ, + NE, + LT, + LE, + GT, + GE, +}; + +std::optional string_dictionary_compare_op(std::string_view name, + bool reverse) { + StringDictionaryCompareOp op; + if (name == "eq") { + op = StringDictionaryCompareOp::EQ; + } else if (name == "ne") { + op = StringDictionaryCompareOp::NE; + } else if (name == "lt") { + op = StringDictionaryCompareOp::LT; + } else if (name == "le") { + op = StringDictionaryCompareOp::LE; + } else if (name == "gt") { + op = StringDictionaryCompareOp::GT; + } else if (name == "ge") { + op = StringDictionaryCompareOp::GE; + } else { + return std::nullopt; + } + if (!reverse || op == StringDictionaryCompareOp::EQ || op == StringDictionaryCompareOp::NE) { + return op; + } + switch (op) { + case StringDictionaryCompareOp::LT: + return StringDictionaryCompareOp::GT; + case StringDictionaryCompareOp::LE: + return StringDictionaryCompareOp::GE; + case StringDictionaryCompareOp::GT: + return StringDictionaryCompareOp::LT; + case StringDictionaryCompareOp::GE: + return StringDictionaryCompareOp::LE; + default: + __builtin_unreachable(); + } +} + +bool string_compare_matches(int comparison, StringDictionaryCompareOp op) { + switch (op) { + case StringDictionaryCompareOp::EQ: + return comparison == 0; + case StringDictionaryCompareOp::NE: + return comparison != 0; + case StringDictionaryCompareOp::LT: + return comparison < 0; + case StringDictionaryCompareOp::LE: + return comparison <= 0; + case StringDictionaryCompareOp::GT: + return comparison > 0; + case StringDictionaryCompareOp::GE: + return comparison >= 0; + } + __builtin_unreachable(); +} + +bool try_apply_string_dictionary_conjunct(size_t block_position, const DataTypePtr& column_type, + const VExprSPtr& root, const IColumn& dictionary, + IColumn::Filter* dictionary_filter) { + const auto fn = std::dynamic_pointer_cast(root); + if (fn == nullptr || (!dictionary.is_column_string() && !dictionary.is_column_string64())) { + return false; + } + const auto slot_literal = expr_zonemap::extract_slot_and_literal(fn->children()); + if (!slot_literal.has_value() || slot_literal->slot_index != block_position || + slot_literal->literal.get_type() != TYPE_STRING || + !remove_nullable(slot_literal->slot_type)->equals(*remove_nullable(column_type)) || + !remove_nullable(slot_literal->literal_type)->equals(*remove_nullable(column_type))) { + return false; + } + const auto op = + string_dictionary_compare_op(fn->function_name(), slot_literal->literal_on_left); + if (!op.has_value()) { + return false; + } + const auto& literal = slot_literal->literal.get(); + const StringRef literal_ref(literal.data(), literal.size()); + for (size_t dictionary_id = 0; dictionary_id < dictionary.size(); ++dictionary_id) { + const int comparison = dictionary.get_data_at(dictionary_id).compare(literal_ref); + (*dictionary_filter)[dictionary_id] &= string_compare_matches(comparison, *op) ? 1 : 0; + } + return true; +} + +Status build_dictionary_entry_filter(size_t block_position, + const ParquetColumnSchema& column_schema, + const VExprContextSPtrs& conjuncts, const IColumn& dictionary, + IColumn::Filter* dictionary_filter, + DictionaryEntryFilterKernel* kernel) { + DORIS_CHECK(dictionary_filter != nullptr); + DORIS_CHECK(kernel != nullptr); + dictionary_filter->clear(); + dictionary_filter->resize_fill(dictionary.size(), 1); + *kernel = DictionaryEntryFilterKernel::GENERIC; + // Block positions are expression slot IDs here; validate the narrowing once so every + // dictionary evaluation path uses the same representable ID. + const int expression_column_id = cast_set(block_position); + const auto typed_data_type = remove_nullable(column_schema.type); + const uint8_t* raw_values = nullptr; + size_t value_width = 0; + if (std::ranges::all_of(conjuncts, + [&](const auto& conjunct) { + return conjunct->root()->can_execute_on_raw_fixed_values( + column_schema.type, expression_column_id); + }) && + get_typed_dictionary_raw_values(typed_data_type->get_primitive_type(), dictionary, + &raw_values, &value_width)) { + // A dictionary is immutable for the row group, so compare its contiguous typed values once + // and reuse the resulting id bitmap for every data page. + for (const auto& conjunct : conjuncts) { + RETURN_IF_ERROR(conjunct->root()->execute_on_raw_fixed_values( + raw_values, dictionary.size(), value_width, column_schema.type, + expression_column_id, dictionary_filter->data())); + } + *kernel = DictionaryEntryFilterKernel::TYPED_FIXED_WIDTH; + return Status::OK(); + } + + if (std::ranges::all_of(conjuncts, [&](const auto& conjunct) { + return try_apply_string_dictionary_conjunct(block_position, column_schema.type, + conjunct->root(), dictionary, + dictionary_filter); + })) { + *kernel = DictionaryEntryFilterKernel::TYPED_STRING; + return Status::OK(); + } + + dictionary_filter->clear(); + dictionary_filter->resize_fill(dictionary.size(), 1); DictionaryEvalContext ctx; auto& slot = ctx.slots - .emplace(static_cast(block_position), + .emplace(expression_column_id, DictionaryEvalContext::SlotDictionary { .data_type = column_schema.type, .values = {}}) .first->second; @@ -1339,12 +1599,13 @@ IColumn::Filter build_dictionary_entry_filter(size_t block_position, dictionary.get(dictionary_id, value); slot.values.clear(); slot.values.push_back(std::move(value)); - dictionary_filter[dictionary_id] = VExprContext::evaluate_dictionary_filter( - conjuncts, ctx) == ZoneMapFilterResult::kNoMatch - ? 0 - : 1; + (*dictionary_filter)[dictionary_id] = + VExprContext::evaluate_dictionary_filter(conjuncts, ctx) == + ZoneMapFilterResult::kNoMatch + ? 0 + : 1; } - return dictionary_filter; + return Status::OK(); } } // namespace @@ -1426,11 +1687,18 @@ Status ParquetScanScheduler::prepare_current_dictionary_filters( // VCompoundPred intentionally evaluates only dictionary-capable children, so residual // predicates still run later on surviving rows. IColumn::Filter dictionary_filter; - DictionaryResidualConjuncts residual_conjuncts; + OwnedExpressionConjuncts residual_conjuncts; { SCOPED_TIMER(_scan_profile.dict_filter_build_time); - dictionary_filter = build_dictionary_entry_filter( - block_position, *column_schema, conjunct_it->second, *dictionary_values); + DictionaryEntryFilterKernel filter_kernel = DictionaryEntryFilterKernel::GENERIC; + RETURN_IF_ERROR(build_dictionary_entry_filter(block_position, *column_schema, + conjunct_it->second, *dictionary_values, + &dictionary_filter, &filter_kernel)); + if (filter_kernel == DictionaryEntryFilterKernel::TYPED_FIXED_WIDTH) { + update_counter_if_not_null(_scan_profile.dict_filter_typed_compare_columns, 1); + } else if (filter_kernel == DictionaryEntryFilterKernel::TYPED_STRING) { + update_counter_if_not_null(_scan_profile.dict_filter_string_compare_columns, 1); + } residual_conjuncts = build_dictionary_residual_conjuncts(conjunct_it->second); } @@ -1471,14 +1739,18 @@ Status ParquetScanScheduler::read_filter_columns(int64_t batch_rows, remember_residual_positions(schedule.remaining_conjuncts); remember_residual_positions(request.delete_conjuncts); const size_t predicate_batch_sequence = _predicate_batch_sequence++; - const bool can_read_predicate_columns_round_by_round = - !schedule.single_column_conjuncts.empty(); + const bool can_read_predicate_columns_round_by_round = schedule.supports_lazy_materialization; auto& read_column_positions = _read_column_positions_scratch; read_column_positions.clear(); read_column_positions.reserve(request.predicate_columns.size()); + auto& materialized_positions = _materialized_predicate_positions_scratch; + materialized_positions.clear(); for (auto& rows : _predicate_column_selection_scratch | std::views::values) { rows.clear(); } + // A generation becomes dirty only when filtering changes SelectionVector. Columns read after + // an all-pass stage already share its coordinates, so rewalking every prior mapping is wasted. + bool predicate_columns_need_alignment = false; auto remember_column_selection = [&](uint32_t position) { auto& rows = _predicate_column_selection_scratch[position]; @@ -1493,6 +1765,8 @@ Status ParquetScanScheduler::read_filter_columns(int64_t batch_rows, auto compact_predicate_columns = [&](bool discard_predicate_only_payload) -> Status { bool compacted = false; int64_t compacted_bytes = 0; + update_counter_if_not_null(_scan_profile.predicate_alignment_columns, + cast_set(read_column_positions.size())); for (const uint32_t position : read_column_positions) { auto& source_rows = _predicate_column_selection_scratch[position]; const auto& old_column = file_block->get_by_position(position).column; @@ -1582,13 +1856,27 @@ Status ParquetScanScheduler::read_filter_columns(int64_t batch_rows, if (dictionary_filter_it != _current_dictionary_filters.end()) { const uint16_t selected_rows_before = *selected_rows; IColumn::Filter compact_filter; + uint16_t new_selected_rows = 0; bool used_filter = false; + const bool predicate_only = request.is_predicate_only(local_id); + // Dictionary ids are sufficient for predicate-only slots; skipping typed survivor + // gathers preserves the block row shape without materializing an unobservable payload. + IColumn* projected_column = predicate_only ? nullptr : column.get(); RETURN_IF_ERROR(column_reader->select_with_dictionary_filter( - *selection, *selected_rows, batch_rows, dictionary_filter_it->second, column, - &compact_filter, &used_filter)); + *selection, *selected_rows, batch_rows, dictionary_filter_it->second, + projected_column, &compact_filter, &new_selected_rows, &used_filter)); if (used_filter) { DORIS_CHECK(compact_filter.size() == selected_rows_before); - const uint16_t new_selected_rows = count_selected_rows(compact_filter); + DORIS_CHECK(new_selected_rows <= selected_rows_before); + update_counter_if_not_null(_scan_profile.dictionary_predicate_direct_batches, 1); + update_counter_if_not_null(_scan_profile.dictionary_predicate_direct_rows, + selected_rows_before); + // The decoder already observes every keep bit while producing compact_filter, so + // reuse its count instead of adding another full filter scan at this boundary. + if (!predicate_only) { + update_counter_if_not_null(_scan_profile.dictionary_predicate_projected_rows, + new_selected_rows); + } const auto filtered_rows = static_cast(selected_rows_before) - static_cast(new_selected_rows); if (conjunct_filtered_rows != nullptr) { @@ -1603,7 +1891,13 @@ Status ParquetScanScheduler::read_filter_columns(int64_t batch_rows, *selected_rows = apply_compact_filter_to_selection(compact_filter, selection, selected_rows_before); } - file_block->replace_by_position(block_position, std::move(column)); + if (predicate_only) { + auto placeholder = column->clone_empty(); + placeholder->insert_many_defaults(*selected_rows); + file_block->replace_by_position(block_position, std::move(placeholder)); + } else { + file_block->replace_by_position(block_position, std::move(column)); + } read_column_positions.push_back(cast_set(block_position)); remember_column_selection(cast_set(block_position)); *used_dictionary_filter = true; @@ -1705,6 +1999,7 @@ Status ParquetScanScheduler::read_filter_columns(int64_t batch_rows, static_cast(new_selected_rows); } if (new_selected_rows != selected_rows_before) { + predicate_columns_need_alignment = true; *selected_rows = can_filter_all ? 0 : apply_compact_filter_to_selection(compact_filter, selection, @@ -1713,16 +2008,16 @@ Status ParquetScanScheduler::read_filter_columns(int64_t batch_rows, return Status::OK(); }; - auto execute_scheduled_dictionary_residual_conjuncts = - [&](const DictionaryResidualConjuncts& conjuncts) -> Status { + auto execute_scheduled_owned_conjuncts = + [&](std::span conjuncts) -> Status { if (conjuncts.empty() || *selected_rows == 0) { return Status::OK(); } const uint16_t selected_rows_before = *selected_rows; IColumn::Filter compact_filter; bool can_filter_all = false; - RETURN_IF_ERROR(execute_compact_dictionary_residual_conjuncts( - conjuncts, selected_rows_before, file_block, &compact_filter, &can_filter_all)); + RETURN_IF_ERROR(execute_compact_owned_conjuncts(conjuncts, selected_rows_before, file_block, + &compact_filter, &can_filter_all)); if (can_filter_all) { compact_filter.resize_fill(selected_rows_before, 0); } @@ -1732,6 +2027,7 @@ Status ParquetScanScheduler::read_filter_columns(int64_t batch_rows, static_cast(new_selected_rows); } if (new_selected_rows != selected_rows_before) { + predicate_columns_need_alignment = true; *selected_rows = can_filter_all ? 0 : apply_compact_filter_to_selection(compact_filter, selection, @@ -1749,13 +2045,13 @@ Status ParquetScanScheduler::read_filter_columns(int64_t batch_rows, return execute_scheduled_conjuncts(conjuncts); }; - auto execute_scheduled_dictionary_residual_conjuncts_with_profile = - [&](const DictionaryResidualConjuncts& conjuncts) -> Status { + auto execute_scheduled_owned_conjuncts_with_profile = + [&](std::span conjuncts) -> Status { if (_scan_profile.predicate_filter_time == nullptr) { - return execute_scheduled_dictionary_residual_conjuncts(conjuncts); + return execute_scheduled_owned_conjuncts(conjuncts); } SCOPED_TIMER(_scan_profile.predicate_filter_time); - return execute_scheduled_dictionary_residual_conjuncts(conjuncts); + return execute_scheduled_owned_conjuncts(conjuncts); }; auto execute_scheduled_delete_conjuncts = [&]() -> Status { @@ -1772,6 +2068,7 @@ Status ParquetScanScheduler::read_filter_columns(int64_t batch_rows, compact_filter.resize_fill(selected_rows_before, 0); } if (can_filter_all || count_selected_rows(compact_filter) != selected_rows_before) { + predicate_columns_need_alignment = true; *selected_rows = can_filter_all ? 0 : apply_compact_filter_to_selection(compact_filter, selection, @@ -1789,6 +2086,7 @@ Status ParquetScanScheduler::read_filter_columns(int64_t batch_rows, RETURN_IF_ERROR(read_predicate_column(column_reader.get(), position_it->second.value(), fid, nullptr, &used_dictionary_filter, &used_fixed_width_filter)); + materialized_positions.insert(position_it->second.value()); } return Status::OK(); }; @@ -1808,8 +2106,18 @@ Status ParquetScanScheduler::read_filter_columns(int64_t batch_rows, // Single-column conjuncts can be evaluated immediately after their column is read. Once // selection shrinks, later predicate columns use ParquetColumnReader::select() so the // reader skips rows already rejected by earlier predicates instead of materializing them. + _ordered_predicate_positions_scratch.clear(); + _ordered_predicate_positions_scratch.reserve(schedule.single_column_conjuncts.size()); + for (const auto& column : request.predicate_columns) { + const size_t position = request.local_positions.at(column.column_id()).value(); + if (schedule.single_column_conjuncts.contains(position)) { + // The request order is the stable cold-start policy until measured costs can + // reorder predicates; unordered-map iteration can defeat an early selective filter. + _ordered_predicate_positions_scratch.push_back(position); + } + } _ordered_predicate_positions_scratch = detail::order_adaptive_predicates( - _predicate_positions_scratch, _predicate_runtime_stats); + _ordered_predicate_positions_scratch, _predicate_runtime_stats); const auto& ordered_positions = _ordered_predicate_positions_scratch; for (size_t order_idx = 0; order_idx < ordered_positions.size(); ++order_idx) { const size_t position = ordered_positions[order_idx]; @@ -1835,16 +2143,20 @@ Status ParquetScanScheduler::read_filter_columns(int64_t batch_rows, RETURN_IF_ERROR(read_predicate_column(reader_it->second.get(), block_position, fid, column_conjuncts, &used_dictionary_filter, &used_fixed_width_filter)); + materialized_positions.insert(block_position); if (*selected_rows != 0 && conjunct_it != schedule.single_column_conjuncts.end()) { if (used_dictionary_filter) { const auto residual_it = _current_dictionary_residual_conjuncts.find(fid); DORIS_CHECK(residual_it != _current_dictionary_residual_conjuncts.end()); - RETURN_IF_ERROR(execute_scheduled_dictionary_residual_conjuncts_with_profile( - residual_it->second)); + RETURN_IF_ERROR( + execute_scheduled_owned_conjuncts_with_profile(residual_it->second)); } else if (!used_fixed_width_filter) { RETURN_IF_ERROR(execute_scheduled_conjuncts_with_profile(conjunct_it->second)); } } + if (*selected_rows != rows_before) { + predicate_columns_need_alignment = true; + } if (sample) { const double cost_per_row = static_cast(MonotonicNanos() - start_ns) / std::max(rows_before, 1); @@ -1866,39 +2178,103 @@ Status ParquetScanScheduler::read_filter_columns(int64_t batch_rows, if (*selected_rows != 0) { continue; } - for (size_t remaining_order_idx = order_idx + 1; - remaining_order_idx < ordered_positions.size(); ++remaining_order_idx) { - const size_t remaining_idx = _predicate_indices_by_position_scratch.at( - ordered_positions[remaining_order_idx]); - const auto remaining_fid = request.predicate_columns[remaining_idx].column_id(); - auto remaining_reader_it = _current_predicate_columns.find(remaining_fid); - DORIS_CHECK(remaining_reader_it != _current_predicate_columns.end()); - RETURN_IF_ERROR(remaining_reader_it->second->skip(batch_rows)); - } return Status::OK(); } return Status::OK(); }; + auto materialize_predicate_positions = [&](const std::vector& positions) -> Status { + for (const size_t position : positions) { + if (materialized_positions.contains(position)) { + continue; + } + const auto index_it = _predicate_indices_by_position_scratch.find(position); + DORIS_CHECK(index_it != _predicate_indices_by_position_scratch.end()); + const auto fid = request.predicate_columns[index_it->second].column_id(); + const auto reader_it = _current_predicate_columns.find(fid); + DORIS_CHECK(reader_it != _current_predicate_columns.end()); + bool used_dictionary_filter = false; + bool used_fixed_width_filter = false; + RETURN_IF_ERROR(read_predicate_column(reader_it->second.get(), position, fid, nullptr, + &used_dictionary_filter, + &used_fixed_width_filter)); + materialized_positions.insert(position); + } + return Status::OK(); + }; + + auto skip_unmaterialized_predicate_columns = [&]() -> Status { + for (const auto& col : request.predicate_columns) { + const auto position_it = request.local_positions.find(col.column_id()); + DORIS_CHECK(position_it != request.local_positions.end()); + if (materialized_positions.contains(position_it->second.value())) { + continue; + } + const auto reader_it = _current_predicate_columns.find(col.column_id()); + DORIS_CHECK(reader_it != _current_predicate_columns.end()); + RETURN_IF_ERROR(reader_it->second->skip(batch_rows)); + } + // Every skipped column has an empty payload in the block. Suppress the caller's + // batch-coordinate filter because there is no materialized batch-sized column left. + *predicate_columns_filtered = true; + return Status::OK(); + }; + auto compact_predicate_columns_with_profile = [&](bool discard_predicate_only_payload) -> Status { + if (!discard_predicate_only_payload && !predicate_columns_need_alignment) { + return Status::OK(); + } const int64_t start_ns = MonotonicNanos(); auto status = compact_predicate_columns(discard_predicate_only_payload); update_counter_if_not_null(_scan_profile.predicate_compaction_time, MonotonicNanos() - start_ns); + if (status.ok()) { + predicate_columns_need_alignment = false; + } return status; }; RETURN_IF_ERROR(read_round_by_round()); - // Single-column expressions only touch the just-read column, so earlier columns can retain - // their own row mappings. Compact only when a later expression needs a shared coordinate - // space; otherwise the final boundary can discard hidden predicate payloads without scanning - // them again. - if (!schedule.remaining_conjuncts.empty()) { + if (*selected_rows == 0) { + RETURN_IF_ERROR(skip_unmaterialized_predicate_columns()); + return compact_predicate_columns_with_profile(true); + } + + // Complex residuals keep their original conjunct order. Materialize only the columns needed + // by the next reachable expression, then compact previously read columns into the same row + // space before evaluating it. This is the scanner-side equivalent of expression-triggered + // lazy columns: a conjunct that rejects the batch prevents later-only columns from decoding. + for (const auto& stage : schedule.remaining_stages) { + RETURN_IF_ERROR(materialize_predicate_positions(stage.required_positions)); RETURN_IF_ERROR(compact_predicate_columns_with_profile(false)); + const OwnedExpressionConjunct stage_conjunct {stage.owner_context, stage.expression}; + RETURN_IF_ERROR(execute_scheduled_owned_conjuncts_with_profile( + std::span(&stage_conjunct, 1))); + if (*selected_rows == 0) { + RETURN_IF_ERROR(skip_unmaterialized_predicate_columns()); + return compact_predicate_columns_with_profile(true); + } } - RETURN_IF_ERROR(execute_scheduled_conjuncts_with_profile(schedule.remaining_conjuncts)); + if (!request.delete_conjuncts.empty()) { + std::set delete_positions; + for (const auto& conjunct : request.delete_conjuncts) { + DORIS_CHECK(conjunct != nullptr && conjunct->root() != nullptr); + conjunct->root()->collect_slot_column_ids(delete_positions); + } + std::vector required_delete_positions; + required_delete_positions.reserve(delete_positions.size()); + for (const int position : delete_positions) { + DORIS_CHECK(position >= 0); + required_delete_positions.push_back(cast_set(position)); + } + if (required_delete_positions.empty() && !_predicate_positions_scratch.empty()) { + // An all-literal equality-delete predicate has no slot dependency, but its hidden + // row-count carrier must still be materialized so the result matches selected_rows. + required_delete_positions.push_back(_predicate_positions_scratch.front()); + } + RETURN_IF_ERROR(materialize_predicate_positions(required_delete_positions)); RETURN_IF_ERROR(compact_predicate_columns_with_profile(false)); } if (_scan_profile.predicate_filter_time == nullptr) { @@ -1907,6 +2283,11 @@ Status ParquetScanScheduler::read_filter_columns(int64_t batch_rows, SCOPED_TIMER(_scan_profile.predicate_filter_time); RETURN_IF_ERROR(execute_scheduled_delete_conjuncts()); } + if (*selected_rows == 0) { + RETURN_IF_ERROR(skip_unmaterialized_predicate_columns()); + return compact_predicate_columns_with_profile(true); + } + RETURN_IF_ERROR(materialize_predicate_positions(_predicate_positions_scratch)); return compact_predicate_columns_with_profile(true); } diff --git a/be/src/format_v2/parquet/parquet_scan.h b/be/src/format_v2/parquet/parquet_scan.h index ac883cbbcbe099..f0e202d99422a8 100644 --- a/be/src/format_v2/parquet/parquet_scan.h +++ b/be/src/format_v2/parquet/parquet_scan.h @@ -58,9 +58,17 @@ struct ParquetScanRange; class NativeParquetMetadata; namespace detail { +struct PredicateConjunctStage { + VExprContextSPtr owner_context; + VExprSPtr expression; + std::vector required_positions; +}; + struct PredicateConjunctSchedule { std::map single_column_conjuncts; VExprContextSPtrs remaining_conjuncts; + std::vector remaining_stages; + bool supports_lazy_materialization = true; }; struct AdaptivePredicateStats { @@ -208,7 +216,7 @@ class ParquetScanScheduler { const detail::PredicateConjunctSchedule& predicate_conjunct_schedule( const format::FileScanRequest& request); std::vector adaptive_predicate_prefetch_columns( - const format::FileScanRequest& request) const; + const format::FileScanRequest& request); Status open_next_row_group(ParquetFileContext& file_context, const std::vector>& file_schema, @@ -305,6 +313,7 @@ class ParquetScanScheduler { detail::PredicateConjunctSchedule _predicate_schedule; std::vector _predicate_positions_scratch; std::unordered_map _predicate_indices_by_position_scratch; + std::unordered_set _materialized_predicate_positions_scratch; std::vector _ordered_predicate_positions_scratch; std::unordered_map> _predicate_column_selection_scratch; diff --git a/be/src/format_v2/parquet/reader/column_reader.cpp b/be/src/format_v2/parquet/reader/column_reader.cpp index db5def75b8c8a7..9eb78718d57e3b 100644 --- a/be/src/format_v2/parquet/reader/column_reader.cpp +++ b/be/src/format_v2/parquet/reader/column_reader.cpp @@ -75,8 +75,8 @@ Status ParquetColumnReader::select(const SelectionVector& selection, uint16_t se } Status ParquetColumnReader::select_with_dictionary_filter(const SelectionVector&, uint16_t, int64_t, - const IColumn::Filter&, MutableColumnPtr&, - IColumn::Filter*, bool*) { + const IColumn::Filter&, IColumn*, + IColumn::Filter*, uint16_t*, bool*) { return Status::NotSupported("Parquet dictionary filter is not implemented for column {}", name()); } diff --git a/be/src/format_v2/parquet/reader/column_reader.h b/be/src/format_v2/parquet/reader/column_reader.h index 6914a5be7d165d..9e33e1a17d968f 100644 --- a/be/src/format_v2/parquet/reader/column_reader.h +++ b/be/src/format_v2/parquet/reader/column_reader.h @@ -58,8 +58,9 @@ class ParquetColumnReader { virtual Status select_with_dictionary_filter(const SelectionVector& selection, uint16_t selected_rows, int64_t batch_rows, const IColumn::Filter& dictionary_filter, - MutableColumnPtr& column, - IColumn::Filter* row_filter, bool* used_filter); + IColumn* projected_column, + IColumn::Filter* row_filter, + uint16_t* survivor_count, bool* used_filter); // Consume batch_rows and evaluate eligible fixed-width values without first constructing a // complete predicate column. Append survivors when projected_column is non-null. Implementations diff --git a/be/src/format_v2/parquet/reader/native/byte_stream_split_decoder.cpp b/be/src/format_v2/parquet/reader/native/byte_stream_split_decoder.cpp index caf3b675e2dfd7..06629a435577be 100644 --- a/be/src/format_v2/parquet/reader/native/byte_stream_split_decoder.cpp +++ b/be/src/format_v2/parquet/reader/native/byte_stream_split_decoder.cpp @@ -22,6 +22,7 @@ #include "core/column/column_fixed_length_object.h" #include "util/byte_stream_split.h" +#include "util/simd/parquet_kernels.h" namespace doris::format::parquet::native { Status ByteStreamSplitDecoder::decode_fixed_values(size_t num_values, @@ -32,8 +33,15 @@ Status ByteStreamSplitDecoder::decode_fixed_values(size_t num_values, } const int64_t stride = static_cast(_data->size / _type_length); _decoded_values.resize(byte_size); - byte_stream_split_decode(reinterpret_cast(_data->data), _type_length, - _offset / _type_length, num_values, stride, _decoded_values.data()); + if (!simd::try_byte_stream_split_decode(reinterpret_cast(_data->data), + _type_length, _offset / _type_length, num_values, + stride, _decoded_values.data())) { + // Unsupported widths, short batches, and non-AVX2 hosts must retain the tuned blocked + // decoder instead of silently falling back to a slower row-by-row transpose. + doris::byte_stream_split_decode(reinterpret_cast(_data->data), _type_length, + _offset / _type_length, num_values, stride, + _decoded_values.data()); + } _offset += byte_size; return consumer.consume(_decoded_values.data(), num_values, static_cast(_type_length)); } @@ -57,9 +65,14 @@ Status ByteStreamSplitDecoder::decode_selected_fixed_values(const ParquetSelecti size_t output = 0; const size_t first_row = _offset / value_width; for (const auto& range : selection.ranges) { - byte_stream_split_decode(reinterpret_cast(_data->data), _type_length, - first_row + range.first, range.count, stride, - _decoded_values.data() + output * value_width); + uint8_t* destination = _decoded_values.data() + output * value_width; + if (!simd::try_byte_stream_split_decode(reinterpret_cast(_data->data), + _type_length, first_row + range.first, range.count, + stride, destination)) { + doris::byte_stream_split_decode(reinterpret_cast(_data->data), + _type_length, first_row + range.first, range.count, + stride, destination); + } output += range.count; } DORIS_CHECK_EQ(output, selection.selected_values); diff --git a/be/src/format_v2/parquet/reader/native/column_chunk_reader.cpp b/be/src/format_v2/parquet/reader/native/column_chunk_reader.cpp index 26afaf15bf4c4c..9625d82557d8c4 100644 --- a/be/src/format_v2/parquet/reader/native/column_chunk_reader.cpp +++ b/be/src/format_v2/parquet/reader/native/column_chunk_reader.cpp @@ -50,6 +50,7 @@ #include "util/bit_util.h" #include "util/block_compression.h" #include "util/cpu_info.h" +#include "util/simd/parquet_kernels.h" #include "util/unaligned.h" namespace cctz { @@ -603,6 +604,13 @@ void expand_nullable_pod_values(ColumnType& column, size_t old_size, size_t comp auto& data = column.get_data(); DORIS_CHECK_EQ(data.size(), old_size + compact_values); data.resize(old_size + selected_nulls.size()); + if constexpr (sizeof(typename ColumnType::value_type) == 4 || + sizeof(typename ColumnType::value_type) == 8) { + simd::expand_nullable_values(reinterpret_cast(data.data() + old_size), + compact_values, selected_nulls.data(), selected_nulls.size(), + sizeof(typename ColumnType::value_type)); + return; + } size_t source = compact_values; for (size_t output = selected_nulls.size(); output > 0;) { --output; @@ -836,7 +844,6 @@ Status ColumnChunkReader::init() { // get the block compression codec RETURN_IF_ERROR(get_block_compression_codec(_metadata.codec, &_block_compress_codec)); _state = INITIALIZED; - RETURN_IF_ERROR(_parse_first_page_header()); return Status::OK(); } @@ -900,30 +907,56 @@ Status ColumnChunkReader::read_levels( } template -Status ColumnChunkReader::_parse_first_page_header() { +Status ColumnChunkReader::_ensure_dictionary_page_loaded() { + if (_dict_checked) { + return Status::OK(); + } + + DORIS_CHECK(_state == INITIALIZED); while (true) { RETURN_IF_ERROR(_page_reader->parse_page_header()); const tparquet::PageHeader* header = nullptr; RETURN_IF_ERROR(_page_reader->get_page_header(&header)); if (header->type == tparquet::PageType::DATA_PAGE || header->type == tparquet::PageType::DATA_PAGE_V2) { - _state = INITIALIZED; - return parse_page_header(); + if constexpr (IN_COLLECTION && OFFSET_INDEX) { + if (header->type == tparquet::PageType::DATA_PAGE && + _page_reader->has_active_offset_index()) { + // V1 nested pages expose row boundaries only in repetition levels, so an + // indexed seek must not skip the first page before those levels are decoded. + _page_reader->discard_offset_index(); + _offset_index = nullptr; + } + } + _dict_checked = true; + return Status::OK(); } if (header->type != tparquet::PageType::DICTIONARY_PAGE) { RETURN_IF_ERROR(_page_reader->skip_auxiliary_page()); - _state = INITIALIZED; continue; } - // the first page maybe directory page even if _metadata.__isset.dictionary_page_offset == false, - // so we should parse the directory page in next_page() RETURN_IF_ERROR(_decode_dict_page()); - // parse the real first data page RETURN_IF_ERROR(_page_reader->dict_next_page()); - _state = INITIALIZED; - // A dictionary is the only non-data page with decoder state. Any following index or - // extension pages are skipped by the same pre-data loop. + // A nested V1 chunk must inspect its first data-page type before indexed seeking can skip + // that page; dictionary discovery alone is not enough to make the OffsetIndex trustworthy. + } +} + +template +Status ColumnChunkReader::load_dictionary_page(bool* has_dict) { + RETURN_IF_ERROR(_ensure_dictionary_page_loaded()); + *has_dict = _has_dict; + return Status::OK(); +} + +template +Status ColumnChunkReader::ensure_first_data_page_parsed() { + if (_first_data_page_parsed) { + return Status::OK(); } + // OffsetIndex row bounds are untrusted until page zero has been reconciled and its declared + // cardinality checked, so no indexed skip may observe them before this one-time parse. + return parse_page_header(); } template @@ -931,6 +964,7 @@ Status ColumnChunkReader::parse_page_header() { if (_state == HEADER_PARSED || _state == DATA_LOADED) { return Status::OK(); } + RETURN_IF_ERROR(_ensure_dictionary_page_loaded()); const tparquet::PageHeader* header = nullptr; while (true) { RETURN_IF_ERROR(_page_reader->parse_page_header()); @@ -996,12 +1030,19 @@ Status ColumnChunkReader::parse_page_header() { if (!active_offset_index) { _chunk_parsed_values += _remaining_num_values; } + _first_data_page_parsed = true; _state = HEADER_PARSED; return Status::OK(); } template Status ColumnChunkReader::next_page() { + if constexpr (OFFSET_INDEX) { + RETURN_IF_ERROR(ensure_first_data_page_parsed()); + } else { + // Load dictionary state before advancing can jump past the physical dictionary page. + RETURN_IF_ERROR(_ensure_dictionary_page_loaded()); + } // Level parsing advances _page_data past the allocation base, so retain explicit ownership // state instead of inferring whether current decoders still reference decompressed storage. _page_uses_decompress_buf = false; @@ -1014,8 +1055,8 @@ Status ColumnChunkReader::next_page() { _decompress_release_pending = false; _decompress_release_threshold = std::numeric_limits::max(); } - _state = INITIALIZED; RETURN_IF_ERROR(_page_reader->next_page()); + _state = INITIALIZED; return Status::OK(); } @@ -1603,8 +1644,10 @@ bool ColumnChunkReader::can_filter_fixed_width_valu } const auto primitive_type = remove_nullable(_field_schema->data_type)->get_primitive_type(); const bool has_identity_width = - (_metadata.type == tparquet::Type::INT32 && primitive_type == TYPE_INT) || - (_metadata.type == tparquet::Type::INT64 && primitive_type == TYPE_BIGINT) || + (_metadata.type == tparquet::Type::INT32 && + (primitive_type == TYPE_INT || primitive_type == TYPE_DECIMAL32)) || + (_metadata.type == tparquet::Type::INT64 && + (primitive_type == TYPE_BIGINT || primitive_type == TYPE_DECIMAL64)) || (_metadata.type == tparquet::Type::FLOAT && primitive_type == TYPE_FLOAT) || (_metadata.type == tparquet::Type::DOUBLE && primitive_type == TYPE_DOUBLE); if (!has_identity_width) { @@ -1704,8 +1747,215 @@ Status ColumnChunkReader::filter_fixed_width_values return Status::OK(); } +namespace { + +template +bool try_filter_and_project_dictionary_values( + const IColumn* typed_dictionary, IColumn* projected_values, + const std::vector& selected_dictionary_indices, + const NullMap& nullable_selection_nulls, const IColumn::Filter& dictionary_filter, + IColumn::Filter* row_filter, size_t* survivor_count) { + if (typed_dictionary == nullptr || projected_values == nullptr) { + return false; + } + const auto* dictionary = check_and_get_column(*typed_dictionary); + auto* projected = check_and_get_column(*projected_values); + if (dictionary == nullptr || projected == nullptr) { + return false; + } + + const auto& dictionary_data = dictionary->get_data(); + auto& projected_data = projected->get_data(); + projected_data.reserve(projected_data.size() + selected_dictionary_indices.size()); + row_filter->reserve(row_filter->size() + nullable_selection_nulls.size()); + size_t physical_row = 0; + size_t survivors = 0; + for (const uint8_t is_null : nullable_selection_nulls) { + bool keep = false; + if (is_null == 0) { + const uint32_t dictionary_id = selected_dictionary_indices[physical_row++]; + // The decoder validates the complete id batch first, preserving atomic output while + // allowing this hot gather loop to use unchecked dictionary lookups. + keep = dictionary_filter[dictionary_id] != 0; + if (keep) { + projected_data.push_back(dictionary_data[dictionary_id]); + ++survivors; + } + } + row_filter->push_back(keep ? 1 : 0); + } + DORIS_CHECK_EQ(physical_row, selected_dictionary_indices.size()); + *survivor_count = survivors; + return true; +} + +bool try_filter_and_project_fixed_width_dictionary( + const IColumn* typed_dictionary, IColumn* projected_values, + const std::vector& selected_dictionary_indices, + const NullMap& nullable_selection_nulls, const IColumn::Filter& dictionary_filter, + IColumn::Filter* row_filter, size_t* survivor_count) { +#define TRY_FIXED_WIDTH_DICTIONARY_COLUMN(ColumnType) \ + if (try_filter_and_project_dictionary_values( \ + typed_dictionary, projected_values, selected_dictionary_indices, \ + nullable_selection_nulls, dictionary_filter, row_filter, survivor_count)) { \ + return true; \ + } + TRY_FIXED_WIDTH_DICTIONARY_COLUMN(ColumnUInt8) + TRY_FIXED_WIDTH_DICTIONARY_COLUMN(ColumnInt8) + TRY_FIXED_WIDTH_DICTIONARY_COLUMN(ColumnInt16) + TRY_FIXED_WIDTH_DICTIONARY_COLUMN(ColumnInt32) + TRY_FIXED_WIDTH_DICTIONARY_COLUMN(ColumnInt64) + TRY_FIXED_WIDTH_DICTIONARY_COLUMN(ColumnInt128) + TRY_FIXED_WIDTH_DICTIONARY_COLUMN(ColumnFloat32) + TRY_FIXED_WIDTH_DICTIONARY_COLUMN(ColumnFloat64) + TRY_FIXED_WIDTH_DICTIONARY_COLUMN(ColumnDate) + TRY_FIXED_WIDTH_DICTIONARY_COLUMN(ColumnDateTime) + TRY_FIXED_WIDTH_DICTIONARY_COLUMN(ColumnDateV2) + TRY_FIXED_WIDTH_DICTIONARY_COLUMN(ColumnDateTimeV2) + TRY_FIXED_WIDTH_DICTIONARY_COLUMN(ColumnTimeV2) + TRY_FIXED_WIDTH_DICTIONARY_COLUMN(ColumnTimeStampTz) + TRY_FIXED_WIDTH_DICTIONARY_COLUMN(ColumnIPv4) + TRY_FIXED_WIDTH_DICTIONARY_COLUMN(ColumnIPv6) + TRY_FIXED_WIDTH_DICTIONARY_COLUMN(ColumnOffset32) + TRY_FIXED_WIDTH_DICTIONARY_COLUMN(ColumnOffset64) + TRY_FIXED_WIDTH_DICTIONARY_COLUMN(ColumnDecimal32) + TRY_FIXED_WIDTH_DICTIONARY_COLUMN(ColumnDecimal64) + TRY_FIXED_WIDTH_DICTIONARY_COLUMN(ColumnDecimal128V2) + TRY_FIXED_WIDTH_DICTIONARY_COLUMN(ColumnDecimal128V3) + TRY_FIXED_WIDTH_DICTIONARY_COLUMN(ColumnDecimal256) +#undef TRY_FIXED_WIDTH_DICTIONARY_COLUMN + return false; +} + +} // namespace + +template +Status ColumnChunkReader::filter_dictionary_indices( + const IColumn::Filter& dictionary_filter, ColumnSelectVector& select_vector, + const IColumn* typed_dictionary, IColumn* projected_values, + ColumnInt32* matched_dictionary_ids, IColumn::Filter* row_filter, size_t* survivor_count, + bool* projected_directly, bool* used_filter) { + DORIS_CHECK(row_filter != nullptr); + DORIS_CHECK(survivor_count != nullptr); + DORIS_CHECK(projected_directly != nullptr); + DORIS_CHECK(used_filter != nullptr); + DORIS_CHECK((typed_dictionary == nullptr) == (projected_values == nullptr)); + *projected_directly = false; + *used_filter = false; + // The top-level reader clears once and owns the final compact filter. Every fully validated + // page fragment appends here so page/range boundaries never require intermediate copies. + *survivor_count = 0; + if (_current_encoding != tparquet::Encoding::RLE_DICTIONARY || _page_decoder == nullptr || + !_page_decoder->has_dictionary()) { + return Status::OK(); + } + if (UNLIKELY(_remaining_num_values < select_vector.num_values())) { + return Status::IOError("Decode too many values in current page"); + } + if (UNLIKELY(dictionary_filter.size() != _page_decoder->dictionary_size())) { + return Status::Corruption("Parquet predicate dictionary has {} entries, expected {}", + dictionary_filter.size(), _page_decoder->dictionary_size()); + } + + ParquetSelection selection; + _nullable_selection_nulls.clear(); + _nullable_selection_nulls.reserve(select_vector.num_values() - select_vector.num_filtered()); + size_t physical_cursor = 0; + auto build_selection = [&]() { + ColumnSelectVector::DataReadType read_type; + while (const size_t run_length = select_vector.get_next_run(&read_type)) { + switch (read_type) { + case ColumnSelectVector::CONTENT: + if (!selection.ranges.empty() && + selection.ranges.back().first + selection.ranges.back().count == + physical_cursor) { + selection.ranges.back().count += run_length; + } else { + selection.ranges.push_back({.first = physical_cursor, .count = run_length}); + } + selection.selected_values += run_length; + _nullable_selection_nulls.resize_fill(_nullable_selection_nulls.size() + run_length, + 0); + physical_cursor += run_length; + break; + case ColumnSelectVector::NULL_DATA: + _nullable_selection_nulls.resize_fill(_nullable_selection_nulls.size() + run_length, + 1); + break; + case ColumnSelectVector::FILTERED_CONTENT: + physical_cursor += run_length; + break; + case ColumnSelectVector::FILTERED_NULL: + break; + } + } + }; + if (select_vector.has_filter()) { + build_selection.template operator()(); + } else { + build_selection.template operator()(); + } + selection.total_values = physical_cursor; + DORIS_CHECK_EQ(selection.total_values, select_vector.num_values() - select_vector.num_nulls()); + DORIS_CHECK_EQ(_nullable_selection_nulls.size(), + select_vector.num_values() - select_vector.num_filtered()); + if (UNLIKELY(_empty_value_section && selection.total_values != 0)) { + return Status::Corruption( + "Parquet definition levels require {} values from an empty value section", + selection.total_values); + } + + _selected_dictionary_indices.clear(); + if (selection.selected_values == 0) { + RETURN_IF_ERROR(_page_decoder->skip_values(selection.total_values)); + } else { + SCOPED_RAW_TIMER(&_chunk_statistics.decode_value_time); + RETURN_IF_ERROR(_page_decoder->decode_selected_dictionary_indices( + selection, &_selected_dictionary_indices)); + } + DORIS_CHECK_EQ(_selected_dictionary_indices.size(), selection.selected_values); + + const bool direct_fixed_width_projection = try_filter_and_project_fixed_width_dictionary( + typed_dictionary, projected_values, _selected_dictionary_indices, + _nullable_selection_nulls, dictionary_filter, row_filter, survivor_count); + auto* matched = + matched_dictionary_ids == nullptr ? nullptr : &matched_dictionary_ids->get_data(); + if (!direct_fixed_width_projection && matched != nullptr) { + matched->reserve(matched->size() + selection.selected_values); + } + if (!direct_fixed_width_projection) { + row_filter->reserve(row_filter->size() + _nullable_selection_nulls.size()); + size_t physical_row = 0; + size_t survivors = 0; + for (const uint8_t is_null : _nullable_selection_nulls) { + bool keep = false; + if (is_null == 0) { + const uint32_t dictionary_id = _selected_dictionary_indices[physical_row++]; + // The complete id batch was validated before this loop, so unchecked bitmap access + // cannot leak partial output for a corrupt page. + keep = dictionary_filter[dictionary_id] != 0; + if (keep && matched != nullptr) { + matched->push_back(cast_set(dictionary_id)); + } + survivors += keep; + } + row_filter->push_back(keep ? 1 : 0); + } + DORIS_CHECK_EQ(physical_row, _selected_dictionary_indices.size()); + *survivor_count = survivors; + } + // Commit page progress only after every external dictionary id has been validated. + _remaining_num_values -= select_vector.num_values(); + *projected_directly = direct_fixed_width_projection; + *used_filter = true; + return Status::OK(); +} + template Status ColumnChunkReader::seek_to_nested_row(size_t left_row) { + if constexpr (IN_COLLECTION && OFFSET_INDEX) { + RETURN_IF_ERROR(ensure_first_data_page_parsed()); + } if constexpr (OFFSET_INDEX) { if (_page_reader->has_active_offset_index()) { while (true) { diff --git a/be/src/format_v2/parquet/reader/native/column_chunk_reader.h b/be/src/format_v2/parquet/reader/native/column_chunk_reader.h index 102366f24949ee..e006f3c59494b3 100644 --- a/be/src/format_v2/parquet/reader/native/column_chunk_reader.h +++ b/be/src/format_v2/parquet/reader/native/column_chunk_reader.h @@ -30,6 +30,7 @@ #include "common/status.h" #include "core/column/column_string.h" +#include "core/column/column_vector.h" #include "core/data_type/data_type.h" #include "core/data_type_serde/parquet_decode_source.h" #include "exprs/vexpr_fwd.h" @@ -204,6 +205,13 @@ class ColumnChunkReader { IColumn::Filter* row_filter, bool* used_filter); bool can_filter_fixed_width_values(const VExprSPtrs& conjuncts, int column_id) const; + Status filter_dictionary_indices(const IColumn::Filter& dictionary_filter, + ColumnSelectVector& select_vector, + const IColumn* typed_dictionary, IColumn* projected_values, + ColumnInt32* matched_dictionary_ids, + IColumn::Filter* row_filter, size_t* survivor_count, + bool* projected_directly, bool* used_filter); + // Get the repetition level decoder of current page. LevelDecoder& rep_level_decoder() { return _rep_level_decoder; } // Get the definition level decoder of current page. @@ -212,7 +220,7 @@ class ColumnChunkReader { level_t max_rep_level() const { return _max_rep_level; } level_t max_def_level() const { return _max_def_level; } - bool has_dict() const { return _has_dict; }; + Status load_dictionary_page(bool* has_dict); // Get page decoder Decoder* get_page_decoder() { return _page_decoder; } @@ -224,6 +232,9 @@ class ColumnChunkReader { // Level decoders may batch-convert unsigned RLE values into Doris' signed level_t. _rep_level_decoder.release_scratch(max_retained_bytes); _def_level_decoder.release_scratch(max_retained_bytes); + if (_selected_dictionary_indices.capacity() * sizeof(uint32_t) > max_retained_bytes) { + std::vector().swap(_selected_dictionary_indices); + } if (_decompress_buf_size > max_retained_bytes) { if (_page_uses_decompress_buf) { // Keep the request until the page boundary because decoders still point into this @@ -246,7 +257,7 @@ class ColumnChunkReader { for (const auto& [encoding, decoder] : _decoders) { bytes += decoder->retained_scratch_bytes(); } - return bytes; + return bytes + _selected_dictionary_indices.capacity() * sizeof(uint32_t); } size_t active_decoder_scratch_bytes() const { @@ -255,7 +266,8 @@ class ColumnChunkReader { return _active_decompress_bytes + (_page_decoder == nullptr ? 0 : _page_decoder->active_scratch_bytes()) + _rep_level_decoder.active_scratch_bytes() + - _def_level_decoder.active_scratch_bytes(); + _def_level_decoder.active_scratch_bytes() + + _selected_dictionary_indices.size() * sizeof(uint32_t); } tparquet::Encoding::type current_encoding() const { return _current_encoding; } @@ -294,6 +306,7 @@ class ColumnChunkReader { size_t page_end_row() const { return _page_reader->end_row(); } + Status ensure_first_data_page_parsed(); Status parse_page_header(); Status next_page(); @@ -334,8 +347,7 @@ class ColumnChunkReader { private: enum ColumnChunkReaderState { NOT_INIT, INITIALIZED, HEADER_PARSED, DATA_LOADED, PAGE_SKIPPED }; - // for check dict page. - Status _parse_first_page_header(); + Status _ensure_dictionary_page_loaded(); Status _decode_dict_page(); void _reserve_decompress_buf(size_t size); @@ -398,6 +410,7 @@ class ColumnChunkReader { Slice _v2_rep_levels; Slice _v2_def_levels; bool _dict_checked = false; + bool _first_data_page_parsed = false; bool _has_dict = false; bool _nested_row_started = false; Decoder* _page_decoder = nullptr; @@ -408,6 +421,7 @@ class ColumnChunkReader { // Plain or Dictionary encoding. If the dictionary grows too big, the encoding will fall back to the plain encoding std::unordered_map> _decoders; NullMap _nullable_selection_nulls; + std::vector _selected_dictionary_indices; ColumnChunkReaderStatistics _chunk_statistics; }; diff --git a/be/src/format_v2/parquet/reader/native/column_reader.cpp b/be/src/format_v2/parquet/reader/native/column_reader.cpp index 592e048846e137..b9c501df9d9eb1 100644 --- a/be/src/format_v2/parquet/reader/native/column_reader.cpp +++ b/be/src/format_v2/parquet/reader/native/column_reader.cpp @@ -38,6 +38,7 @@ #include "core/data_type/data_type_nullable.h" #include "core/data_type/data_type_struct.h" #include "core/data_type/define_primitive_type.h" +#include "core/data_type_serde/parquet_decode_source.h" #include "exprs/vexpr.h" #include "format_v2/parquet/native_schema_desc.h" #include "format_v2/parquet/reader/native/column_chunk_reader.h" @@ -75,6 +76,36 @@ size_t retained_set_bytes(const std::unordered_set& values) { return values.bucket_count() * sizeof(void*) + values.size() * sizeof(size_t); } +bool append_compact_int_dictionary(const IColumn& dictionary, const uint32_t* indices, size_t count, + IColumn* destination) { + auto* destination_int = check_and_get_column(*destination); + const auto* dictionary_int = check_and_get_column(dictionary); + if (destination_int == nullptr || dictionary_int == nullptr) { + return false; + } + if (count == 0) { + return true; + } + auto& output = destination_int->get_data(); + const auto& values = dictionary_int->get_data(); + const size_t old_size = output.size(); + output.resize(old_size + count); + int32_t* dst = output.data() + old_size; + size_t row = 0; + // A tiny Parquet INT dictionary normally remains in L1. This V1-style scalar gather avoids + // AVX gather setup and keeps the survivor-id loop compact enough for the compiler to unroll. + for (; row + 4 <= count; row += 4) { + dst[row] = values[indices[row]]; + dst[row + 1] = values[indices[row + 1]]; + dst[row + 2] = values[indices[row + 2]]; + dst[row + 3] = values[indices[row + 3]]; + } + for (; row < count; ++row) { + dst[row] = values[indices[row]]; + } + return true; +} + Status validate_decimal_physical_type(const NativeFieldSchema& field, int precision, int scale) { if (precision <= 0 || scale < 0 || scale > precision) { return Status::Corruption("Parquet decimal field {} has invalid precision {} and scale {}", @@ -1137,10 +1168,10 @@ Status ScalarColumnReader::read_fixed_width_filter( int64_t right_row = 0; if constexpr (OFFSET_INDEX == false) { RETURN_IF_ERROR(_chunk_reader->parse_page_header()); - right_row = _chunk_reader->page_end_row(); } else { - right_row = _chunk_reader->page_end_row(); + RETURN_IF_ERROR(_chunk_reader->ensure_first_data_page_parsed()); } + right_row = _chunk_reader->page_end_row(); RowRanges read_ranges; _generate_read_ranges(RowRange {_current_row_index, right_row}, &read_ranges); if (read_ranges.count() == 0) { @@ -1183,6 +1214,138 @@ Status ScalarColumnReader::read_fixed_width_filter( return Status::OK(); } +template +Status ScalarColumnReader::_read_dictionary_filter_values( + size_t num_values, const IColumn::Filter& dictionary_filter, FilterMap& filter_map, + const IColumn* typed_dictionary, IColumn* projected_values, + ColumnInt32* matched_dictionary_ids, IColumn::Filter* row_filter, size_t* survivor_count, + bool* projected_directly) { + DORIS_CHECK(row_filter != nullptr); + DORIS_CHECK(survivor_count != nullptr); + DORIS_CHECK(projected_directly != nullptr); + _null_run_lengths.clear(); + if (_chunk_reader->max_def_level() > 0) { + LevelDecoder& def_decoder = _chunk_reader->def_level_decoder(); + size_t has_read = 0; + bool prev_is_null = true; + while (has_read < num_values) { + level_t def_level = -1; + const size_t loop_read = def_decoder.get_next_run(&def_level, num_values - has_read); + if (loop_read == 0) { + return Status::Corruption( + "Parquet definition level stream ended while filtering dictionary ids"); + } + const bool is_null = def_level < _field_schema->definition_level; + if (!(prev_is_null ^ is_null)) { + _null_run_lengths.emplace_back(0); + } + size_t remaining = loop_read; + while (remaining > USHRT_MAX) { + _null_run_lengths.emplace_back(USHRT_MAX); + _null_run_lengths.emplace_back(0); + remaining -= USHRT_MAX; + } + _null_run_lengths.emplace_back(cast_set(remaining)); + prev_is_null = is_null; + has_read += loop_read; + } + } else { + size_t remaining = num_values; + while (remaining > USHRT_MAX) { + _null_run_lengths.emplace_back(USHRT_MAX); + _null_run_lengths.emplace_back(0); + remaining -= USHRT_MAX; + } + _null_run_lengths.emplace_back(cast_set(remaining)); + } + RETURN_IF_ERROR(_select_vector.init(_null_run_lengths, num_values, nullptr, &filter_map, + _filter_map_index)); + _filter_map_index += num_values; + bool used_filter = false; + RETURN_IF_ERROR(_chunk_reader->filter_dictionary_indices( + dictionary_filter, _select_vector, typed_dictionary, projected_values, + matched_dictionary_ids, row_filter, survivor_count, projected_directly, &used_filter)); + // Pure-dictionary chunks are prevalidated before definition levels are consumed. + DORIS_CHECK(used_filter); + return Status::OK(); +} + +template +Status ScalarColumnReader::read_dictionary_filter( + const IColumn::Filter& dictionary_filter, FilterMap& filter_map, size_t batch_size, + const IColumn* typed_dictionary, IColumn* projected_values, + ColumnInt32* matched_dictionary_ids, IColumn::Filter* row_filter, size_t* survivor_count, + size_t* read_rows, bool* eof, bool* projected_directly, bool* used_filter) { + DORIS_CHECK(row_filter != nullptr); + DORIS_CHECK(survivor_count != nullptr); + DORIS_CHECK(read_rows != nullptr); + DORIS_CHECK(eof != nullptr); + DORIS_CHECK(projected_directly != nullptr); + DORIS_CHECK(used_filter != nullptr); + *survivor_count = 0; + *read_rows = 0; + *projected_directly = false; + *used_filter = false; + if (_in_nested || dictionary_filter.empty()) { + return Status::OK(); + } + + int64_t right_row = 0; + if constexpr (OFFSET_INDEX == false) { + RETURN_IF_ERROR(_chunk_reader->parse_page_header()); + right_row = _chunk_reader->page_end_row(); + } else { + right_row = _chunk_reader->page_end_row(); + } + RowRanges read_ranges; + _generate_read_ranges(RowRange {_current_row_index, right_row}, &read_ranges); + if (read_ranges.count() == 0) { + _current_row_index = right_row; + } else { + RETURN_IF_ERROR(_chunk_reader->parse_page_header()); + RETURN_IF_ERROR(_chunk_reader->load_page_data_idempotent()); + if (_chunk_reader->current_encoding() != tparquet::Encoding::RLE_DICTIONARY) { + return Status::OK(); + } + size_t has_read = 0; + for (size_t idx = 0; idx < read_ranges.range_size(); ++idx) { + const auto range = read_ranges.get_range(idx); + const size_t skip_values = range.from() - _current_row_index; + RETURN_IF_ERROR(_skip_values(skip_values)); + _current_row_index += skip_values; + const size_t values = + std::min(static_cast(range.to() - range.from()), batch_size - has_read); + size_t fragment_survivors = 0; + bool fragment_projected_directly = false; + RETURN_IF_ERROR(_read_dictionary_filter_values( + values, dictionary_filter, filter_map, typed_dictionary, projected_values, + matched_dictionary_ids, row_filter, &fragment_survivors, + &fragment_projected_directly)); + if (has_read != 0) { + DORIS_CHECK_EQ(*projected_directly, fragment_projected_directly); + } + *projected_directly = fragment_projected_directly; + *survivor_count += fragment_survivors; + has_read += values; + *read_rows += values; + _current_row_index += values; + if (has_read == batch_size) { + break; + } + } + } + + if (right_row == _current_row_index) { + if (!_chunk_reader->has_next_page()) { + *eof = true; + } else { + RETURN_IF_ERROR(_chunk_reader->next_page()); + } + } + *used_filter = true; + return Status::OK(); +} + template Status ScalarColumnReader::read_column_levels(FilterMap& filter_map, size_t batch_size, @@ -1259,10 +1422,8 @@ Status ScalarColumnReader::read_column_levels(Filte } template -Result -ScalarColumnReader::materialize_dictionary_values( - const ColumnInt32* dict_column, const DataTypePtr& target_type) { - DORIS_CHECK(dict_column != nullptr); +Status ScalarColumnReader::_ensure_typed_dictionary( + const DataTypePtr& target_type) { DORIS_CHECK(target_type != nullptr); Decoder* dictionary_decoder = _chunk_reader->dictionary_decoder(); DORIS_CHECK(dictionary_decoder != nullptr); @@ -1284,7 +1445,7 @@ ScalarColumnReader::materialize_dictionary_values( auto status = dictionary_serde->read_parquet_dictionary( *_materialization_state.typed_dictionary, *dictionary_decoder, dictionary_context); if (!status.ok()) { - return ResultError(std::move(status)); + return status; } DORIS_CHECK_EQ(_materialization_state.typed_dictionary->size(), dictionary_decoder->dictionary_size()); @@ -1293,8 +1454,24 @@ ScalarColumnReader::materialize_dictionary_values( ++_dictionary_materialization_count; #endif } + return Status::OK(); +} + +template +Status ScalarColumnReader::prepare_typed_dictionary( + const DataTypePtr& target_type, const IColumn** dictionary) { + DORIS_CHECK(dictionary != nullptr); + RETURN_IF_ERROR(_ensure_typed_dictionary(target_type)); + *dictionary = _materialization_state.typed_dictionary.get(); + return Status::OK(); +} - auto result = _materialization_state.typed_dictionary->clone_empty(); +template +Status ScalarColumnReader::append_dictionary_values( + const ColumnInt32* dict_column, const DataTypePtr& target_type, IColumn* destination) { + DORIS_CHECK(dict_column != nullptr); + DORIS_CHECK(destination != nullptr); + RETURN_IF_ERROR(_ensure_typed_dictionary(target_type)); const auto& source_indices = dict_column->get_data(); auto& indices = _materialization_state.dictionary_indices; indices.resize(source_indices.size()); @@ -1302,20 +1479,59 @@ ScalarColumnReader::materialize_dictionary_values( if (UNLIKELY(source_indices[row] < 0 || static_cast(source_indices[row]) >= _materialization_state.typed_dictionary->size())) { - return ResultError(Status::Corruption( + return Status::Corruption( "Parquet dictionary index {} at row {} exceeds dictionary size {}", - source_indices[row], row, _materialization_state.typed_dictionary->size())); + source_indices[row], row, _materialization_state.typed_dictionary->size()); } indices[row] = static_cast(source_indices[row]); } - result->insert_indices_from(*_materialization_state.typed_dictionary, indices.data(), - indices.data() + indices.size()); + + IColumn* values_destination = destination; + ColumnNullable* nullable_destination = check_and_get_column(*destination); + if (nullable_destination != nullptr) { + values_destination = &nullable_destination->get_nested_column(); + } + // The compact id array contains survivors only. INT uses the V1-style L1-friendly gather; + // other fixed-width types use SIMD and variable-width types retain generic typed insertion. + if (!append_compact_int_dictionary(*_materialization_state.typed_dictionary, indices.data(), + indices.size(), values_destination) && + !try_simd_insert_parquet_dictionary_indices(*values_destination, + *_materialization_state.typed_dictionary, + indices.data(), indices.size())) { + values_destination->insert_indices_from(*_materialization_state.typed_dictionary, + indices.data(), indices.data() + indices.size()); + } + if (nullable_destination != nullptr) { + auto& null_map = nullable_destination->get_null_map_data(); + null_map.resize_fill(null_map.size() + indices.size(), 0); + } + return Status::OK(); +} + +template +Result +ScalarColumnReader::materialize_dictionary_values( + const ColumnInt32* dict_column, const DataTypePtr& target_type) { + DORIS_CHECK(target_type != nullptr); + auto result = remove_nullable(target_type)->create_column(); + auto status = append_dictionary_values(dict_column, target_type, result.get()); + if (!status.ok()) { + return ResultError(std::move(status)); + } return result; } template Result ScalarColumnReader::dictionary_values( const DataTypePtr& target_type) { + bool has_dict = false; + auto status = _chunk_reader->load_dictionary_page(&has_dict); + if (!status.ok()) { + return ResultError(std::move(status)); + } + if (!has_dict) { + return ResultError(Status::NotSupported("Parquet column has no reusable dictionary")); + } Decoder* dictionary_decoder = _chunk_reader->dictionary_decoder(); if (dictionary_decoder == nullptr || dictionary_decoder->dictionary_size() == 0) { return ResultError(Status::NotSupported("Parquet column has no reusable dictionary")); @@ -1331,15 +1547,6 @@ Result ScalarColumnReader::dictio return materialize_dictionary_values(ids.get(), target_type); } -template -Status ScalarColumnReader::_try_load_dict_page(bool* loaded, - bool* has_dict) { - // _chunk_reader init will load first page header to check whether has dict page - *loaded = true; - *has_dict = _chunk_reader->has_dict(); - return Status::OK(); -} - template Status ScalarColumnReader::read_column_data( ColumnPtr& doris_column, const DataTypePtr& type, @@ -1382,10 +1589,10 @@ Status ScalarColumnReader::read_column_data( int64_t right_row = 0; if constexpr (OFFSET_INDEX == false) { RETURN_IF_ERROR(_chunk_reader->parse_page_header()); - right_row = _chunk_reader->page_end_row(); } else { - right_row = _chunk_reader->page_end_row(); + RETURN_IF_ERROR(_chunk_reader->ensure_first_data_page_parsed()); } + right_row = _chunk_reader->page_end_row(); do { // generate the row ranges that should be read diff --git a/be/src/format_v2/parquet/reader/native/column_reader.h b/be/src/format_v2/parquet/reader/native/column_reader.h index 6ea2787f9f0211..c6b844e47f74d3 100644 --- a/be/src/format_v2/parquet/reader/native/column_reader.h +++ b/be/src/format_v2/parquet/reader/native/column_reader.h @@ -201,6 +201,24 @@ class ColumnReader { return Status::OK(); } + virtual Status read_dictionary_filter(const IColumn::Filter&, FilterMap&, size_t, + const IColumn*, IColumn*, ColumnInt32*, + IColumn::Filter* row_filter, size_t* survivor_count, + size_t* read_rows, bool* eof, bool* projected_directly, + bool* used_filter) { + DORIS_CHECK(row_filter != nullptr); + DORIS_CHECK(survivor_count != nullptr); + DORIS_CHECK(read_rows != nullptr); + DORIS_CHECK(eof != nullptr); + DORIS_CHECK(projected_directly != nullptr); + DORIS_CHECK(used_filter != nullptr); + *survivor_count = 0; + *read_rows = 0; + *projected_directly = false; + *used_filter = false; + return Status::OK(); + } + // Consume a nested batch while retaining only definition/repetition levels. This is used when // schema evolution makes every projected STRUCT child synthetic: the parent still needs one // physical leaf's shape, but decoding that leaf's strings or other payload would be wasted. @@ -212,6 +230,12 @@ class ColumnReader { throw Exception( Status::FatalError("Method materialize_dictionary_values is not supported")); } + virtual Status append_dictionary_values(const ColumnInt32*, const DataTypePtr&, IColumn*) { + return Status::NotSupported("Appending parquet dictionary values is not supported"); + } + virtual Status prepare_typed_dictionary(const DataTypePtr&, const IColumn**) { + return Status::NotSupported("Typed parquet dictionary is not supported"); + } virtual Result dictionary_values(const DataTypePtr& target_type) { return ResultError(Status::NotSupported("Parquet dictionary values are not supported")); } @@ -283,10 +307,20 @@ class ScalarColumnReader : public ColumnReader { FilterMap& filter_map, size_t batch_size, IColumn* projected_column, IColumn::Filter* row_filter, size_t* read_rows, bool* eof, bool* used_filter) override; + Status read_dictionary_filter(const IColumn::Filter& dictionary_filter, FilterMap& filter_map, + size_t batch_size, const IColumn* typed_dictionary, + IColumn* projected_values, ColumnInt32* matched_dictionary_ids, + IColumn::Filter* row_filter, size_t* survivor_count, + size_t* read_rows, bool* eof, bool* projected_directly, + bool* used_filter) override; Status read_column_levels(FilterMap& filter_map, size_t batch_size, size_t* read_rows, bool* eof) override; Result materialize_dictionary_values(const ColumnInt32* dict_column, const DataTypePtr& target_type) override; + Status append_dictionary_values(const ColumnInt32* dict_column, const DataTypePtr& target_type, + IColumn* destination) override; + Status prepare_typed_dictionary(const DataTypePtr& target_type, + const IColumn** dictionary) override; Result dictionary_values(const DataTypePtr& target_type) override; const std::vector& get_rep_level() const override { return _rep_levels; } const std::vector& get_def_level() const override { return _def_levels; } @@ -408,9 +442,17 @@ class ScalarColumnReader : public ColumnReader { Status _read_fixed_width_filter_values(size_t num_values, const VExprSPtrs& conjuncts, int column_id, FilterMap& filter_map, IColumn* projected_column, IColumn::Filter* row_filter); + Status _read_dictionary_filter_values(size_t num_values, + const IColumn::Filter& dictionary_filter, + FilterMap& filter_map, const IColumn* typed_dictionary, + IColumn* projected_values, + ColumnInt32* matched_dictionary_ids, + IColumn::Filter* row_filter, size_t* survivor_count, + bool* projected_directly); Status _read_nested_column(ColumnPtr& doris_column, const DataTypePtr& type, FilterMap& filter_map, size_t batch_size, size_t* read_rows, bool* eof, bool is_dict_filter); + Status _ensure_typed_dictionary(const DataTypePtr& target_type); Status _try_load_dict_page(bool* loaded, bool* has_dict); }; diff --git a/be/src/format_v2/parquet/reader/native/decoder.cpp b/be/src/format_v2/parquet/reader/native/decoder.cpp index 5f6a978317bde7..c068cafaf0da5c 100644 --- a/be/src/format_v2/parquet/reader/native/decoder.cpp +++ b/be/src/format_v2/parquet/reader/native/decoder.cpp @@ -122,7 +122,10 @@ Status Decoder::get_decoder(tparquet::Type::type type, tparquet::Encoding::type switch (encoding) { case tparquet::Encoding::PLAIN: return create_plain_decoder(type, decoder); + case tparquet::Encoding::PLAIN_DICTIONARY: case tparquet::Encoding::RLE_DICTIONARY: + // PLAIN_DICTIONARY is the legacy page enum for the same RLE/bit-packed id stream; accepting + // it here keeps every dictionary decode entry point consistent with ColumnChunkReader. return create_dictionary_decoder(type, decoder); case tparquet::Encoding::RLE: if (type != tparquet::Type::BOOLEAN) { diff --git a/be/src/format_v2/parquet/reader/native/decoder.h b/be/src/format_v2/parquet/reader/native/decoder.h index 379176d5e459e0..628a34ac97675e 100644 --- a/be/src/format_v2/parquet/reader/native/decoder.h +++ b/be/src/format_v2/parquet/reader/native/decoder.h @@ -182,6 +182,12 @@ class BaseDictDecoder : public Decoder { std::vector* indices) override { DORIS_CHECK(indices != nullptr); const size_t num_dictionary_values = dictionary_size(); + if (_is_fragmented_selection(selection)) { + RETURN_IF_ERROR(_decode_fragmented_selection(selection, num_dictionary_values)); + indices->assign(_skip_indices.begin(), + _skip_indices.begin() + selection.selected_values); + return Status::OK(); + } indices->resize(selection.selected_values); size_t cursor = 0; size_t output = 0; @@ -223,6 +229,10 @@ class BaseDictDecoder : public Decoder { Status decode_selected_dictionary_values(const ParquetSelection& selection, ParquetDictionaryValueConsumer& consumer) override { const size_t num_dictionary_values = dictionary_size(); + if (_is_fragmented_selection(selection)) { + RETURN_IF_ERROR(_decode_fragmented_selection(selection, num_dictionary_values)); + return consumer.consume_indices(_skip_indices.data(), selection.selected_values); + } size_t cursor = 0; for (const auto& range : selection.ranges) { DORIS_CHECK(range.first >= cursor); @@ -246,6 +256,57 @@ class BaseDictDecoder : public Decoder { size_t active_scratch_bytes() const override { return _skip_indices.size() * sizeof(uint32_t); } protected: + static bool _is_fragmented_selection(const ParquetSelection& selection) { + constexpr size_t MIN_FRAGMENTED_RANGES = 8; + constexpr size_t MAX_AVERAGE_RANGE_VALUES = 4; + constexpr size_t MAX_DECODE_EXPANSION = 8; + return selection.ranges.size() >= MIN_FRAGMENTED_RANGES && selection.selected_values != 0 && + selection.total_values / selection.selected_values <= MAX_DECODE_EXPANSION && + selection.selected_values <= selection.ranges.size() * MAX_AVERAGE_RANGE_VALUES; + } + + Status _decode_fragmented_selection(const ParquetSelection& selection, + size_t num_dictionary_values) { + // Decode and validate the page batch once when predicate survivors alternate in tiny runs. + // Walking each range separately turns one RLE batch into millions of decoder calls for + // low-cardinality predicates such as TPC-DS quantity buckets. + _skip_indices.resize(selection.total_values); + const auto decoded = _index_batch_decoder->GetBatch( + _skip_indices.data(), cast_set(selection.total_values)); + if (UNLIKELY(decoded != selection.total_values)) { + return Status::IOError("Can't read enough Parquet dictionary indices"); + } + if (UNLIKELY(!dictionary_indices_in_bounds(_skip_indices.data(), selection.total_values, + num_dictionary_values))) { + for (size_t row = 0; row < selection.total_values; ++row) { + if (_skip_indices[row] < num_dictionary_values) { + continue; + } + return Status::Corruption( + "Parquet dictionary index {} at row {} exceeds dictionary size {}", + _skip_indices[row], row, num_dictionary_values); + } + } + size_t output = 0; + constexpr size_t MAX_INLINE_COPY_VALUES = 4; + for (const auto& range : selection.ranges) { + DORIS_CHECK(range.first + range.count <= selection.total_values); + // Alternating predicates mostly produce one-row spans; inline tiny forward copies so + // range compaction does not replace decoder calls with equally numerous libc calls. + if (range.count <= MAX_INLINE_COPY_VALUES) { + for (size_t row = 0; row < range.count; ++row) { + _skip_indices[output + row] = _skip_indices[range.first + row]; + } + } else { + memmove(_skip_indices.data() + output, _skip_indices.data() + range.first, + range.count * sizeof(uint32_t)); + } + output += range.count; + } + DORIS_CHECK_EQ(output, selection.selected_values); + return Status::OK(); + } + Status skip_values(size_t num_values) override { return _decode_and_validate_skipped(num_values, 0, dictionary_size()); } diff --git a/be/src/format_v2/parquet/reader/native/delta_bit_pack_decoder.h b/be/src/format_v2/parquet/reader/native/delta_bit_pack_decoder.h index db9efecaced470..94e248c5fe506b 100644 --- a/be/src/format_v2/parquet/reader/native/delta_bit_pack_decoder.h +++ b/be/src/format_v2/parquet/reader/native/delta_bit_pack_decoder.h @@ -39,6 +39,7 @@ #include "format_v2/parquet/reader/native/decoder.h" #include "util/bit_stream_utils.h" #include "util/bit_stream_utils.inline.h" +#include "util/simd/parquet_kernels.h" #include "util/slice.h" namespace doris::format::parquet::native { @@ -787,13 +788,9 @@ Status DeltaBitPackDecoder::_get_internal(T* buffer, uint32_t num_values, return Status::IOError("Get batch EOF"); } } - for (int j = 0; j < values_decode; ++j) { - // Addition between min_delta, packed int and last_value should be treated as - // unsigned addition. Overflow is as expected. - buffer[i + j] = static_cast(_min_delta) + static_cast(buffer[i + j]) + - static_cast(_last_value); - _last_value = buffer[i + j]; - } + // Parquet defines this recurrence with wrapping integer arithmetic. The SIMD kernel keeps + // the same unsigned-overflow invariant while resolving the prefix dependency in batches. + simd::delta_decode(buffer + i, values_decode, _min_delta, &_last_value); _values_remaining_current_mini_block -= values_decode; i += values_decode; } diff --git a/be/src/format_v2/parquet/reader/native_column_reader.cpp b/be/src/format_v2/parquet/reader/native_column_reader.cpp index 233b37c096e300..99c0154eea2e35 100644 --- a/be/src/format_v2/parquet/reader/native_column_reader.cpp +++ b/be/src/format_v2/parquet/reader/native_column_reader.cpp @@ -143,20 +143,6 @@ void collect_projected_ids(const ParquetColumnSchema& schema, } } -Status append_non_null_dictionary_values(MutableColumnPtr& target, MutableColumnPtr values) { - DORIS_CHECK(target); - DORIS_CHECK(values); - const size_t value_count = values->size(); - if (auto* nullable = check_and_get_column(*target); nullable != nullptr) { - nullable->get_nested_column().insert_range_from(*values, 0, value_count); - auto& null_map = nullable->get_null_map_data(); - null_map.resize_fill(null_map.size() + value_count, 0); - return Status::OK(); - } - target->insert_range_from(*values, 0, value_count); - return Status::OK(); -} - } // namespace NativeColumnReader::NativeColumnReader(const ParquetColumnSchema& schema, @@ -395,6 +381,75 @@ Status NativeColumnReader::read_with_fixed_width_filter(int64_t rows, const uint return Status::OK(); } +Status NativeColumnReader::read_with_dictionary_filter( + int64_t rows, const uint8_t* filter_data, bool filter_all, + const IColumn::Filter& dictionary_filter, const IColumn* typed_dictionary, + IColumn* projected_values, ColumnInt32* matched_dictionary_ids, IColumn::Filter* row_filter, + int64_t* survivor_count, int64_t* rows_read, bool* projected_directly, bool* used_filter) { + DORIS_CHECK(rows >= 0); + DORIS_CHECK(row_filter != nullptr); + DORIS_CHECK(survivor_count != nullptr); + DORIS_CHECK(rows_read != nullptr); + DORIS_CHECK(projected_directly != nullptr); + DORIS_CHECK(used_filter != nullptr); + row_filter->clear(); + *survivor_count = 0; + *rows_read = 0; + *projected_directly = false; + *used_filter = false; + if (rows == 0) { + return Status::OK(); + } + + native::FilterMap filter; + RETURN_IF_ERROR(filter.init(filter_data, static_cast(rows), filter_all)); + _native_reader->reset_filter_map_index(); + bool eof = false; + int64_t consecutive_empty_calls = 0; + while (*rows_read < rows && !eof) { + size_t loop_rows = 0; + size_t loop_survivors = 0; + bool loop_projected_directly = false; + bool loop_used = false; + RETURN_IF_ERROR(_native_reader->read_dictionary_filter( + dictionary_filter, filter, static_cast(rows - *rows_read), typed_dictionary, + projected_values, matched_dictionary_ids, row_filter, &loop_survivors, &loop_rows, + &eof, &loop_projected_directly, &loop_used)); + if (!loop_used) { + if (UNLIKELY(*rows_read != 0)) { + return Status::Corruption( + "Parquet dictionary predicate encoding changed after {} rows for column {}", + *rows_read, _name); + } + row_filter->clear(); + return Status::OK(); + } + if (*rows_read != 0) { + DORIS_CHECK_EQ(*projected_directly, loop_projected_directly); + } + *projected_directly = loop_projected_directly; + *survivor_count += cast_set(loop_survivors); + if (loop_rows == 0 && !eof) { + if (++consecutive_empty_calls > _row_group_rows + 1) { + return Status::Corruption( + "Native parquet dictionary predicate made no progress for column {}", + _name); + } + continue; + } + consecutive_empty_calls = 0; + *rows_read += static_cast(loop_rows); + } + if (*rows_read != rows) { + return Status::Corruption( + "Native parquet dictionary predicate returned {} rows, expected {} for {}", + *rows_read, rows, _name); + } + *used_filter = true; + release_batch_scratch_if_needed(); + return Status::OK(); +} + void NativeColumnReader::release_batch_scratch_if_needed() { // PLAIN predicate batches bypass materialization but share the same persistent decoder tree, // so both read paths must advance the retained-capacity aging clock. @@ -512,24 +567,78 @@ Status NativeColumnReader::select(const SelectionVector& selection, uint16_t sel return Status::OK(); } -Status NativeColumnReader::select_with_dictionary_filter(const SelectionVector& selection, - uint16_t selected_rows, int64_t batch_rows, - const IColumn::Filter& dictionary_filter, - MutableColumnPtr& column, - IColumn::Filter* row_filter, - bool* used_filter) { +Status NativeColumnReader::select_with_dictionary_filter( + const SelectionVector& selection, uint16_t selected_rows, int64_t batch_rows, + const IColumn::Filter& dictionary_filter, IColumn* projected_column, + IColumn::Filter* row_filter, uint16_t* survivor_count, bool* used_filter) { DORIS_CHECK(row_filter != nullptr); + DORIS_CHECK(survivor_count != nullptr); DORIS_CHECK(used_filter != nullptr); RETURN_IF_ERROR(validate_selected_span(batch_rows)); *used_filter = false; + *survivor_count = 0; row_filter->clear(); if (!_dictionary_filter_enabled) { return Status::OK(); } *used_filter = true; + row_filter->reserve(selected_rows); const uint8_t* filter_data = nullptr; RETURN_IF_ERROR(selection.materialize_filter(selected_rows, batch_rows, &filter_data)); + ColumnInt32* direct_matched_ids = nullptr; + const IColumn* typed_dictionary = nullptr; + IColumn* projected_values = projected_column; + ColumnNullable* projected_nullable = nullptr; + if (projected_column != nullptr) { + if (!_matched_dictionary_ids) { + _matched_dictionary_ids = ColumnInt32::create(); + } + _matched_dictionary_ids->clear(); + direct_matched_ids = check_and_get_column(*_matched_dictionary_ids); + DORIS_CHECK(direct_matched_ids != nullptr); + RETURN_IF_ERROR(_native_reader->prepare_typed_dictionary(_type, &typed_dictionary)); + projected_nullable = check_and_get_column(*projected_column); + if (projected_nullable != nullptr) { + projected_values = &projected_nullable->get_nested_column(); + } + } + int64_t direct_rows_read = 0; + int64_t direct_survivor_count = 0; + bool projected_directly = false; + bool direct_filter_used = false; + RETURN_IF_ERROR(read_with_dictionary_filter( + batch_rows, filter_data, selected_rows == 0, dictionary_filter, typed_dictionary, + projected_values, direct_matched_ids, row_filter, &direct_survivor_count, + &direct_rows_read, &projected_directly, &direct_filter_used)); + if (direct_filter_used) { + advance_selected_span(direct_rows_read); + *survivor_count = cast_set(direct_survivor_count); + if (projected_column != nullptr) { + if (projected_directly) { + DORIS_CHECK(direct_matched_ids->empty()); + if (projected_nullable != nullptr) { + auto& null_map = projected_nullable->get_null_map_data(); + null_map.resize_fill(null_map.size() + *survivor_count, 0); + } + if (_profile.dictionary_predicate_fused_projected_rows != nullptr) { + COUNTER_UPDATE(_profile.dictionary_predicate_fused_projected_rows, + *survivor_count); + } + } else { + DORIS_CHECK_EQ(direct_matched_ids->size(), *survivor_count); + RETURN_IF_ERROR(_native_reader->append_dictionary_values(direct_matched_ids, _type, + projected_column)); + } + } + if (_profile.reader_select_rows != nullptr) { + COUNTER_UPDATE(_profile.reader_select_rows, selected_rows); + } + update_reader_read_rows(*survivor_count); + update_reader_skip_rows(batch_rows - *survivor_count); + return Status::OK(); + } + const bool nullable = _type->is_nullable(); DataTypePtr id_type = std::make_shared(); if (nullable) { @@ -560,13 +669,18 @@ Status NativeColumnReader::select_with_dictionary_filter(const SelectionVector& } DORIS_CHECK(ids != nullptr); - if (!_matched_dictionary_ids) { - _matched_dictionary_ids = ColumnInt32::create(); + ColumnInt32::Container* matched_ids = nullptr; + if (projected_column != nullptr) { + if (!_matched_dictionary_ids) { + _matched_dictionary_ids = ColumnInt32::create(); + } + _matched_dictionary_ids->clear(); + matched_ids = &assert_cast(*_matched_dictionary_ids).get_data(); + matched_ids->reserve(selected_rows); } - _matched_dictionary_ids->clear(); - auto& matched_ids = assert_cast(*_matched_dictionary_ids).get_data(); row_filter->reserve(selected_rows); const auto& id_data = ids->get_data(); + size_t fallback_survivor_count = 0; for (size_t row = 0; row < selected_rows; ++row) { bool keep = false; if (null_map == nullptr || (*null_map)[row] == 0) { @@ -579,22 +693,27 @@ Status NativeColumnReader::select_with_dictionary_filter(const SelectionVector& } keep = dictionary_filter[static_cast(dictionary_id)] != 0; if (keep) { - matched_ids.push_back(dictionary_id); + ++fallback_survivor_count; + if (matched_ids != nullptr) { + matched_ids->push_back(dictionary_id); + } } } row_filter->push_back(keep ? 1 : 0); } - const auto* matched_id_column = check_and_get_column(*_matched_dictionary_ids); - DORIS_CHECK(matched_id_column != nullptr); - auto matched_values = - DORIS_TRY(_native_reader->materialize_dictionary_values(matched_id_column, _type)); - RETURN_IF_ERROR(append_non_null_dictionary_values(column, std::move(matched_values))); + if (projected_column != nullptr) { + const auto* matched_id_column = check_and_get_column(*_matched_dictionary_ids); + DORIS_CHECK(matched_id_column != nullptr); + RETURN_IF_ERROR(_native_reader->append_dictionary_values(matched_id_column, _type, + projected_column)); + } if (_profile.reader_select_rows != nullptr) { COUNTER_UPDATE(_profile.reader_select_rows, selected_rows); } - update_reader_read_rows(cast_set(matched_ids.size())); - update_reader_skip_rows(batch_rows - cast_set(matched_ids.size())); + *survivor_count = cast_set(fallback_survivor_count); + update_reader_read_rows(*survivor_count); + update_reader_skip_rows(batch_rows - *survivor_count); return Status::OK(); } diff --git a/be/src/format_v2/parquet/reader/native_column_reader.h b/be/src/format_v2/parquet/reader/native_column_reader.h index 1d92735bfa19d3..d4f3df59c804d4 100644 --- a/be/src/format_v2/parquet/reader/native_column_reader.h +++ b/be/src/format_v2/parquet/reader/native_column_reader.h @@ -86,8 +86,8 @@ class NativeColumnReader final : public ParquetColumnReader { Status select_with_dictionary_filter(const SelectionVector& selection, uint16_t selected_rows, int64_t batch_rows, const IColumn::Filter& dictionary_filter, - MutableColumnPtr& column, IColumn::Filter* row_filter, - bool* used_filter) override; + IColumn* projected_column, IColumn::Filter* row_filter, + uint16_t* survivor_count, bool* used_filter) override; Status select_with_fixed_width_filter(const SelectionVector& selection, uint16_t selected_rows, int64_t batch_rows, const VExprSPtrs& conjuncts, int column_id, IColumn* projected_column, @@ -116,6 +116,13 @@ class NativeColumnReader final : public ParquetColumnReader { const VExprSPtrs& conjuncts, int column_id, IColumn* projected_column, IColumn::Filter* row_filter, int64_t* rows_read, bool* used_filter); + Status read_with_dictionary_filter(int64_t rows, const uint8_t* filter_data, bool filter_all, + const IColumn::Filter& dictionary_filter, + const IColumn* typed_dictionary, IColumn* projected_values, + ColumnInt32* matched_dictionary_ids, + IColumn::Filter* row_filter, int64_t* survivor_count, + int64_t* rows_read, bool* projected_directly, + bool* used_filter); void release_batch_scratch_if_needed(); int64_t sync_native_profile(); void record_page_fragments(int64_t page_fragments); diff --git a/be/src/format_v2/parquet/selection_vector.h b/be/src/format_v2/parquet/selection_vector.h index ab2bf93c785edc..033478875fad95 100644 --- a/be/src/format_v2/parquet/selection_vector.h +++ b/be/src/format_v2/parquet/selection_vector.h @@ -67,6 +67,7 @@ class SelectionVector { _owned.clear(); _data = data; _size = count; + _identity = data == nullptr; ++_generation; } @@ -77,6 +78,7 @@ class SelectionVector { for (size_t idx = 0; idx < count; ++idx) { _data[idx] = static_cast(idx); } + _identity = true; ++_generation; } @@ -84,6 +86,7 @@ class SelectionVector { _owned.clear(); _data = nullptr; _size = 0; + _identity = true; ++_generation; } @@ -91,7 +94,13 @@ class SelectionVector { bool is_set() const { return _data != nullptr; } - Index* data() { return _data; } + Index* data() { + // A mutable pointer can change indices without set_index(), so identity can no longer be + // proven until resize() rebuilds it. This keeps the O(1) dense fast path conservative. + _identity = false; + ++_generation; + return _data; + } const Index* data() const { return _data; } @@ -104,11 +113,21 @@ class SelectionVector { void set_index(size_t idx, Index value) { _data[idx] = value; + if (value != idx) { + _identity = false; + } ++_generation; } Status materialize_filter(size_t count, int64_t batch_rows, const uint8_t** filter) const { DORIS_CHECK(filter != nullptr); + if (batch_rows >= 0 && std::cmp_equal(count, batch_rows) && _identity && + (_data == nullptr || count <= _size)) { + // A proven identity selection is equivalent to no FilterMap. Returning nullptr avoids + // constructing and rescanning one dense byte per source row. + *filter = nullptr; + return Status::OK(); + } RETURN_IF_ERROR(verify(count, batch_rows)); if (_filter_generation != _generation || _filter_count != count || _filter_batch_rows != batch_rows) { @@ -161,6 +180,7 @@ class SelectionVector { std::vector _owned; Index* _data = nullptr; size_t _size = 0; + bool _identity = true; uint64_t _generation = 0; mutable std::vector _filter; mutable uint64_t _filter_generation = std::numeric_limits::max(); diff --git a/be/src/format_v2/table/hudi_reader.cpp b/be/src/format_v2/table/hudi_reader.cpp index 271f14a7b483eb..ee24d0f9ad7d02 100644 --- a/be/src/format_v2/table/hudi_reader.cpp +++ b/be/src/format_v2/table/hudi_reader.cpp @@ -125,27 +125,6 @@ void HudiHybridReader::set_batch_size(size_t batch_size) { } } -Status HudiHybridReader::append_conjuncts(const VExprContextSPtrs& conjuncts) { - // The wrapper snapshot initializes future children, while every existing child needs the same - // late RF immediately so active and later reused splits keep identical predicate ownership. - const size_t owned_count = - _appended_table_reader_owned_conjunct_count.value_or(conjuncts.size()); - RETURN_IF_ERROR(format::TableReader::append_conjuncts(conjuncts)); - if (_native_reader != nullptr) { - RETURN_IF_ERROR(_native_reader->append_conjuncts_with_ownership(conjuncts, owned_count)); - } - if (_jni_reader != nullptr) { - RETURN_IF_ERROR(_jni_reader->append_conjuncts_with_ownership(conjuncts, owned_count)); - } - return Status::OK(); -} - -const format::MaterializedBlockStats& HudiHybridReader::last_materialized_block_stats() const { - // FileScannerV2 budgets cooperative work from the child that actually materialized the block. - return _current_split_reader != nullptr ? _current_split_reader->last_materialized_block_stats() - : format::TableReader::last_materialized_block_stats(); -} - int64_t HudiHybridReader::condition_cache_hit_count() const { // Keep the wrapper count cumulative across native/JNI dispatch so scanner-level delta // accounting neither loses a child hit nor observes a counter reset on a split switch. @@ -197,7 +176,6 @@ Status HudiHybridReader::_init_child_reader(format::TableReader* reader, RETURN_IF_ERROR(reader->init({ .projected_columns = _projected_columns, .conjuncts = std::move(conjuncts), - .table_reader_owned_conjunct_count = _table_reader_owned_conjunct_count, .format = file_format, .scan_params = _scan_params, .io_ctx = _io_ctx, diff --git a/be/src/format_v2/table/hudi_reader.h b/be/src/format_v2/table/hudi_reader.h index c13ac1215d72ee..dbb6f5e8231043 100644 --- a/be/src/format_v2/table/hudi_reader.h +++ b/be/src/format_v2/table/hudi_reader.h @@ -65,8 +65,6 @@ class HudiHybridReader final : public format::TableReader { Status abort_split() override; Status close() override; void set_batch_size(size_t batch_size) override; - Status append_conjuncts(const VExprContextSPtrs& conjuncts) override; - const format::MaterializedBlockStats& last_materialized_block_stats() const override; int64_t condition_cache_hit_count() const override; #ifdef BE_TEST diff --git a/be/src/format_v2/table/iceberg_position_delete_sys_table_reader.cpp b/be/src/format_v2/table/iceberg_position_delete_sys_table_reader.cpp index 6ab67caa77da03..b2b37c1bf09226 100644 --- a/be/src/format_v2/table/iceberg_position_delete_sys_table_reader.cpp +++ b/be/src/format_v2/table/iceberg_position_delete_sys_table_reader.cpp @@ -163,12 +163,6 @@ Status IcebergPositionDeleteSysTableV2Reader::prepare_split( const format::SplitReadOptions& options) { RETURN_IF_ERROR(close()); RETURN_IF_ERROR(format::TableReader::prepare_split(options)); - if (current_split_pruned()) { - return Status::OK(); - } - // This synthetic reader has no physical schema where a predicate can be localized, so every - // split predicate must run after its system-table columns have been materialized. - RETURN_IF_ERROR(_prepare_all_conjuncts_as_remaining()); // The inner delete-file reader has distinct counters, so the outer preparation can safely // contain its cache miss/open work without re-entering the same RuntimeProfile timer. SCOPED_TIMER(_profile.total_timer); @@ -181,7 +175,6 @@ Status IcebergPositionDeleteSysTableV2Reader::prepare_split( Status IcebergPositionDeleteSysTableV2Reader::get_block(Block* block, bool* eos) { SCOPED_TIMER(_profile.total_timer); SCOPED_TIMER(_profile.exec_timer); - _reset_materialized_block_stats(); DORIS_CHECK(block != nullptr); DORIS_CHECK(eos != nullptr); DORIS_CHECK(block->columns() == _projected_columns.size()); @@ -199,19 +192,9 @@ Status IcebergPositionDeleteSysTableV2Reader::get_block(Block* block, bool* eos) return Status::OK(); } + size_t read_rows = 0; if (_delete_file_kind == DeleteFileKind::DELETION_VECTOR) { - size_t read_rows = 0; - RETURN_IF_ERROR(_append_deletion_vector_block(block, &read_rows, eos)); - if (read_rows > 0) { - _record_materialized_block_stats(*block, read_rows); - RETURN_IF_ERROR(_filter_remaining_conjuncts(block, &read_rows)); - } - if (read_rows == 0) { - // Yield after one deletion-vector batch so cancellation and Scanner row budgets are - // observed even when residual predicates reject every synthesized row. - block->clear_column_data(_projected_columns.size()); - } - return Status::OK(); + return _append_deletion_vector_block(block, &read_rows, eos); } DORIS_CHECK(_position_reader != nullptr); @@ -225,18 +208,8 @@ Status IcebergPositionDeleteSysTableV2Reader::get_block(Block* block, bool* eos) RETURN_IF_ERROR(_position_reader->get_block(&delete_block, &position_reader_eof)); const size_t delete_rows = delete_block.rows(); if (delete_rows > 0) { - size_t read_rows = 0; RETURN_IF_ERROR( _append_position_delete_block(block, delete_block, delete_rows, &read_rows)); - _record_materialized_block_stats(*block, read_rows); - RETURN_IF_ERROR(_filter_remaining_conjuncts(block, &read_rows)); - if (read_rows == 0) { - // A filtered materialized batch is still progress; return it to Scanner instead of - // consuming an unbounded number of position-delete batches in this call. - block->clear_column_data(_projected_columns.size()); - *eos = false; - return Status::OK(); - } *eos = false; return Status::OK(); } @@ -343,12 +316,10 @@ Status IcebergPositionDeleteSysTableV2Reader::_init_position_delete_reader() { static constexpr const char* kPositionReaderProfile = "IcebergPositionDeleteFileReader"; if (_position_reader_profile == nullptr) { - _position_reader_profile = _scanner_profile->get_child(kPositionReaderProfile); - if (_position_reader_profile == nullptr) { - // The outer system-table reader calls the inner reader synchronously. Giving both the - // same profile would nest identical counter pointers and double-count every timer. - _position_reader_profile = _scanner_profile->create_child(kPositionReaderProfile); - } + // The nested reader needs a distinct profile to avoid double-counting its timers. Split + // readers share the scanner profile and initialize concurrently, so lookup and creation + // must also be atomic to preserve the unique child-name invariant. + _position_reader_profile = _scanner_profile->get_or_create_child(kPositionReaderProfile); } _position_reader = std::make_unique(); RETURN_IF_ERROR(_position_reader->init({ diff --git a/be/src/format_v2/table/paimon_reader.cpp b/be/src/format_v2/table/paimon_reader.cpp index d815ef81c9b424..a3f4092a470263 100644 --- a/be/src/format_v2/table/paimon_reader.cpp +++ b/be/src/format_v2/table/paimon_reader.cpp @@ -104,6 +104,14 @@ Status PaimonHybridReader::prepare_split(const format::SplitReadOptions& options // timer around the first native or JNI child and double-count that initialization. RETURN_IF_ERROR(_ensure_current_split_reader(options)); DORIS_CHECK(_current_split_reader != nullptr); + if (!_is_jni_split(options.current_range)) { + auto native_options = options; + // Legacy FE plans wrap native files in FORMAT_JNI; normalize the child contract so the + // physical reader does not overwrite its recovered Parquet/ORC format with that wrapper. + RETURN_IF_ERROR( + _to_file_format(options.current_range, &native_options.current_split_format)); + return _current_split_reader->prepare_split(native_options); + } return _current_split_reader->prepare_split(options); } @@ -152,27 +160,6 @@ void PaimonHybridReader::set_batch_size(size_t batch_size) { } } -Status PaimonHybridReader::append_conjuncts(const VExprContextSPtrs& conjuncts) { - // The wrapper snapshot initializes future children, while every existing child needs the same - // late RF immediately so active and later reused splits keep identical predicate ownership. - const size_t owned_count = - _appended_table_reader_owned_conjunct_count.value_or(conjuncts.size()); - RETURN_IF_ERROR(format::TableReader::append_conjuncts(conjuncts)); - if (_native_reader != nullptr) { - RETURN_IF_ERROR(_native_reader->append_conjuncts_with_ownership(conjuncts, owned_count)); - } - if (_jni_reader != nullptr) { - RETURN_IF_ERROR(_jni_reader->append_conjuncts_with_ownership(conjuncts, owned_count)); - } - return Status::OK(); -} - -const format::MaterializedBlockStats& PaimonHybridReader::last_materialized_block_stats() const { - // FileScannerV2 budgets cooperative work from the child that actually materialized the block. - return _current_split_reader != nullptr ? _current_split_reader->last_materialized_block_stats() - : format::TableReader::last_materialized_block_stats(); -} - int64_t PaimonHybridReader::condition_cache_hit_count() const { // Both children survive split switches, so the wrapper must publish their cumulative totals; // returning only the active child would make FileScannerV2's monotonic delta go backwards. @@ -199,7 +186,10 @@ Status PaimonHybridReader::_ensure_current_split_reader(const format::SplitReadO } else { format::FileFormat file_format; RETURN_IF_ERROR(_to_file_format(options.current_range, &file_format)); - DCHECK(options.current_split_format == file_format); + // Old FE plans encoded a native file as FORMAT_JNI without paimon_split and carried the + // physical format only in paimon_params.file_format. + DCHECK(options.current_split_format == file_format || + options.current_split_format == format::FileFormat::JNI); DCHECK(file_format == format::FileFormat::PARQUET || file_format == format::FileFormat::ORC); if (_native_reader == nullptr) { @@ -227,7 +217,6 @@ Status PaimonHybridReader::_init_child_reader(format::TableReader* reader, RETURN_IF_ERROR(reader->init({ .projected_columns = _projected_columns, .conjuncts = std::move(conjuncts), - .table_reader_owned_conjunct_count = _table_reader_owned_conjunct_count, .format = file_format, .scan_params = _scan_params, .io_ctx = _io_ctx, @@ -258,16 +247,30 @@ Status PaimonHybridReader::_clone_conjuncts(VExprContextSPtrs* conjuncts) const } bool PaimonHybridReader::_is_jni_split(const TFileRangeDesc& range) { - return range.__isset.table_format_params && range.table_format_params.__isset.paimon_params && - range.table_format_params.paimon_params.__isset.reader_type && - range.table_format_params.paimon_params.reader_type == TPaimonReaderType::PAIMON_JNI; + if (!range.__isset.table_format_params || !range.table_format_params.__isset.paimon_params) { + return false; + } + const auto& params = range.table_format_params.paimon_params; + return params.__isset.paimon_split && + (!params.__isset.reader_type || params.reader_type == TPaimonReaderType::PAIMON_JNI); } Status PaimonHybridReader::_to_file_format(const TFileRangeDesc& range, format::FileFormat* file_format) { DORIS_CHECK(file_format != nullptr); - const auto format_type = + auto format_type = range.__isset.format_type ? range.format_type : TFileFormatType::FORMAT_PARQUET; + // JNI splits also carry file_format metadata; only a split without paimon_split can use + // FORMAT_JNI as the legacy encoding of a native file. + if (format_type == TFileFormatType::FORMAT_JNI && !_is_jni_split(range) && + range.__isset.table_format_params && range.table_format_params.__isset.paimon_params) { + const auto& params = range.table_format_params.paimon_params; + if (params.__isset.file_format && params.file_format == "orc") { + format_type = TFileFormatType::FORMAT_ORC; + } else if (params.__isset.file_format && params.file_format == "parquet") { + format_type = TFileFormatType::FORMAT_PARQUET; + } + } switch (format_type) { case TFileFormatType::FORMAT_PARQUET: *file_format = format::FileFormat::PARQUET; diff --git a/be/src/format_v2/table/paimon_reader.h b/be/src/format_v2/table/paimon_reader.h index b4f076fc6e4878..823fa6540d280c 100644 --- a/be/src/format_v2/table/paimon_reader.h +++ b/be/src/format_v2/table/paimon_reader.h @@ -71,8 +71,6 @@ class PaimonHybridReader final : public format::TableReader { Status abort_split() override; Status close() override; void set_batch_size(size_t batch_size) override; - Status append_conjuncts(const VExprContextSPtrs& conjuncts) override; - const format::MaterializedBlockStats& last_materialized_block_stats() const override; int64_t condition_cache_hit_count() const override; #ifdef BE_TEST diff --git a/be/src/format_v2/table_reader.cpp b/be/src/format_v2/table_reader.cpp index a073f01f237506..8a5da295fddd4a 100644 --- a/be/src/format_v2/table_reader.cpp +++ b/be/src/format_v2/table_reader.cpp @@ -90,6 +90,8 @@ std::string file_format_to_string(FileFormat format) { return "NATIVE"; case FileFormat::ARROW: return "ARROW"; + case FileFormat::WAL: + return "WAL"; } return "UNKNOWN"; } @@ -678,8 +680,6 @@ Status TableReader::init(TableReadOptions&& options) { ADD_CHILD_TIMER_WITH_LEVEL(_scanner_profile, "PrepareSplitTime", table_profile, 1); _profile.finalize_timer = ADD_CHILD_TIMER_WITH_LEVEL(_scanner_profile, "FinalizeBlockTime", table_profile, 1); - _profile.residual_filter_timer = ADD_CHILD_TIMER_WITH_LEVEL( - _scanner_profile, "ResidualFilterTime", table_profile, 1); _profile.create_reader_timer = ADD_CHILD_TIMER_WITH_LEVEL(_scanner_profile, "CreateReaderTime", table_profile, 1); _profile.pushdown_agg_timer = @@ -723,9 +723,6 @@ Status TableReader::init(TableReadOptions&& options) { _push_down_count_columns = options.push_down_count_columns; _initial_condition_cache_digest = options.condition_cache_digest; _condition_cache_digest = _initial_condition_cache_digest; - _table_reader_owned_conjunct_count = - options.table_reader_owned_conjunct_count.value_or(options.conjuncts.size()); - DORIS_CHECK_LE(_table_reader_owned_conjunct_count, options.conjuncts.size()); _projected_columns = std::move(options.projected_columns); if (supports_iceberg_scan_semantics_v1(_scan_params)) { for (auto& projected_column : _projected_columns) { @@ -739,64 +736,7 @@ Status TableReader::init(TableReadOptions&& options) { } _system_properties = create_system_properties(_scan_params); _mapper_options.mode = TableColumnMappingMode::BY_NAME; - return _replace_conjuncts(options.conjuncts); -} - -Status TableReader::_prepare_conjunct(const VExprContextSPtr& source, VExprContextSPtr* prepared) { - DORIS_CHECK(source != nullptr); - DORIS_CHECK(source->root() != nullptr); - DORIS_CHECK(prepared != nullptr); - VExprSPtr root; - RETURN_IF_ERROR(clone_table_expr_tree(source->root(), &root)); - auto conjunct = VExprContext::create_shared(std::move(root)); - RETURN_IF_ERROR(conjunct->prepare(_runtime_state, RowDescriptor {})); - RETURN_IF_ERROR(conjunct->open(_runtime_state)); - *prepared = std::move(conjunct); - return Status::OK(); -} - -Status TableReader::_replace_conjuncts(const VExprContextSPtrs& conjuncts) { - VExprContextSPtrs prepared; - prepared.reserve(conjuncts.size()); - for (const auto& source : conjuncts) { - VExprContextSPtr conjunct; - RETURN_IF_ERROR(_prepare_conjunct(source, &conjunct)); - prepared.push_back(std::move(conjunct)); - } - _conjuncts = std::move(prepared); - return Status::OK(); -} - -Status TableReader::append_conjuncts_with_ownership(const VExprContextSPtrs& conjuncts, - size_t table_reader_owned_conjunct_count) { - DORIS_CHECK(!_appended_table_reader_owned_conjunct_count.has_value()); - _appended_table_reader_owned_conjunct_count = table_reader_owned_conjunct_count; - auto status = append_conjuncts(conjuncts); - _appended_table_reader_owned_conjunct_count.reset(); - return status; -} - -Status TableReader::append_conjuncts(const VExprContextSPtrs& conjuncts) { - const size_t owned_count = - _appended_table_reader_owned_conjunct_count.value_or(conjuncts.size()); - DORIS_CHECK_LE(owned_count, conjuncts.size()); - // Once Scanner owns a suffix, later predicates cannot be inserted into the TableReader-owned - // prefix without reordering them ahead of that stateful/error-preserving barrier. - DORIS_CHECK(owned_count == 0 || _table_reader_owned_conjunct_count == _conjuncts.size()); - for (size_t conjunct_index = 0; conjunct_index < conjuncts.size(); ++conjunct_index) { - const auto& source = conjuncts[conjunct_index]; - VExprContextSPtr conjunct; - RETURN_IF_ERROR(_prepare_conjunct(source, &conjunct)); - _conjuncts.push_back(conjunct); - if (conjunct_index < owned_count) { - ++_table_reader_owned_conjunct_count; - } - if (_current_task != nullptr && conjunct_index < owned_count) { - // The active reader has already fixed its localized predicate set. Appended runtime - // filters must remain residual until the next split rebuilds its FileScanRequest. - _remaining_conjuncts.push_back(std::move(conjunct)); - } - } + _conjuncts = std::move(options.conjuncts); return Status::OK(); } @@ -804,27 +744,20 @@ Status TableReader::_build_table_filters_from_conjuncts() { _table_filters.clear(); _constant_pruning_safe_filter_count = 0; bool in_safe_prefix = true; - for (size_t conjunct_index = 0; conjunct_index < _conjuncts.size(); ++conjunct_index) { - const auto& conjunct = _conjuncts[conjunct_index]; + for (const auto& conjunct : _conjuncts) { DORIS_CHECK(conjunct != nullptr); DORIS_CHECK(conjunct->root() != nullptr); // `_table_filters` omits expressions without slot references, but such an expression still // occupies a position in the row-level conjunct order. Record how many localized filters // precede the first unsafe original conjunct so constant pruning cannot jump over a - // slotless non-deterministic/error-preserving barrier. An unsafe predicate is either kept - // on TableReader's post-materialization path by a standalone caller or carried only for - // analysis when FileScannerV2 owns the ordered suffix. - if (in_safe_prefix && !is_safe_to_pre_execute(conjunct)) { + // slotless non-deterministic/error-preserving barrier. Unsafe predicates remain solely on + // Scanner's original row-level path because localizing a clone would execute their state + // twice with independent state. + if (in_safe_prefix && !_is_safe_to_pre_execute(conjunct)) { in_safe_prefix = false; } - const size_t filters_before = _table_filters.size(); RETURN_IF_ERROR( build_table_filters_from_conjunct(conjunct, _runtime_state, &_table_filters)); - for (size_t filter_index = filters_before; filter_index < _table_filters.size(); - ++filter_index) { - _table_filters[filter_index].source_conjunct_index = conjunct_index; - _table_filters[filter_index].can_localize = in_safe_prefix; - } if (in_safe_prefix) { _constant_pruning_safe_filter_count = _table_filters.size(); } @@ -832,59 +765,6 @@ Status TableReader::_build_table_filters_from_conjuncts() { return Status::OK(); } -Status TableReader::_prepare_all_conjuncts_as_remaining() { - // Expression contexts carry mutable state (for example sequence/stateful functions). Select - // from the TableReader-owned contexts instead of reopening clones for every split. - _remaining_conjuncts.assign( - _conjuncts.begin(), - _conjuncts.begin() + cast_set(_table_reader_owned_conjunct_count)); - return Status::OK(); -} - -Status TableReader::_prepare_remaining_conjuncts( - const FilterLocalizationResult& localization_result) { - DORIS_CHECK(localization_result.localized_filters.size() == _table_filters.size()); - std::vector localized_conjuncts(_conjuncts.size(), false); - for (size_t filter_index = 0; filter_index < _table_filters.size(); ++filter_index) { - if (!localization_result.localized_filters[filter_index]) { - continue; - } - const size_t source_index = _table_filters[filter_index].source_conjunct_index; - DORIS_CHECK(source_index < localized_conjuncts.size()); - localized_conjuncts[source_index] = true; - } - - _remaining_conjuncts.clear(); - for (size_t conjunct_index = 0; conjunct_index < _table_reader_owned_conjunct_count; - ++conjunct_index) { - if (localized_conjuncts[conjunct_index]) { - continue; - } - _remaining_conjuncts.push_back(_conjuncts[conjunct_index]); - } - return Status::OK(); -} - -Status TableReader::_filter_remaining_conjuncts(Block* block, size_t* rows) { - DORIS_CHECK(block != nullptr); - DORIS_CHECK(rows != nullptr); - if (*rows == 0 || _remaining_conjuncts.empty()) { - return Status::OK(); - } - SCOPED_TIMER(_profile.residual_filter_timer); - const size_t rows_before_filter = *rows; - auto status = VExprContext::filter_block(_remaining_conjuncts, block, block->columns()); - if (!status.ok() && _format == FileFormat::ORC) { - status.prepend("Orc row reader nextBatch failed. reason = "); - } - RETURN_IF_ERROR(status); - *rows = block->columns() == 0 ? rows_before_filter : block->rows(); - if (_io_ctx != nullptr) { - _io_ctx->predicate_filtered_rows += rows_before_filter - *rows; - } - return Status::OK(); -} - Status TableReader::_open_local_filter_exprs(const FileScanRequest& file_request) { RowDescriptor row_desc; for (const auto& conjunct : file_request.conjuncts) { @@ -1125,9 +1005,6 @@ Status TableReader::prepare_split(const SplitReadOptions& options) { SCOPED_TIMER(_profile.total_timer); SCOPED_TIMER(_profile.prepare_split_timer); _current_split_pruned = false; - // Predicate localization belongs to the physical schema of one split. Clear the previous - // ownership before any early return so a pruned or failed split cannot leak it to the next one. - _remaining_conjuncts.clear(); _all_runtime_filters_applied_for_split = options.all_runtime_filters_applied; _condition_cache_digest_covers_current_split = options.condition_cache_digest.has_value(); if (options.condition_cache_digest.has_value()) { @@ -1141,7 +1018,7 @@ Status TableReader::prepare_split(const SplitReadOptions& options) { _condition_cache_digest = _initial_condition_cache_digest; } if (options.conjuncts.has_value()) { - RETURN_IF_ERROR(_replace_conjuncts(*options.conjuncts)); + _conjuncts = *options.conjuncts; } // Update to current split format to handle ORC/PARQUET files in one table. _format = options.current_split_format; @@ -1211,7 +1088,7 @@ Status TableReader::_evaluate_partition_prune_conjuncts(const VExprContextSPtrs& // Keep only the safe prefix of the original conjunct order. If an unsafe conjunct is // skipped, a later predicate could prune the split before the unsafe one reaches its // normal row-level evaluation point. - if (!is_safe_to_pre_execute(conjunct)) { + if (!_is_safe_to_pre_execute(conjunct)) { break; } std::set global_indices; @@ -1224,8 +1101,9 @@ Status TableReader::_evaluate_partition_prune_conjuncts(const VExprContextSPtrs& return false; } const auto& column = _projected_columns[index.value()]; - return column.is_partition_key && - find_partition_value(column, _partition_values) != nullptr; + // Identity-partition metadata is a split constant even when the same source column + // must remain file-backed for data written under another evolved partition spec. + return find_partition_value(column, _partition_values) != nullptr; }); if (partition_only) { partition_conjuncts.push_back(conjunct); @@ -1247,7 +1125,7 @@ Status TableReader::_evaluate_partition_prune_conjuncts(const VExprContextSPtrs& can_filter_all); } -bool TableReader::is_safe_to_pre_execute(const VExprContextSPtr& conjunct) { +bool TableReader::_is_safe_to_pre_execute(const VExprContextSPtr& conjunct) { DORIS_CHECK(conjunct != nullptr); DORIS_CHECK(conjunct->root() != nullptr); const auto root = conjunct->root(); @@ -1266,11 +1144,9 @@ Status TableReader::_build_partition_prune_block(Block* block) const { for (const auto& column : _projected_columns) { DORIS_CHECK(column.type != nullptr); ColumnPtr value_column = column.type->create_column_const_with_default_value(1); - if (column.is_partition_key) { - const auto* partition_value = find_partition_value(column, _partition_values); - if (partition_value != nullptr) { - value_column = column.type->create_column_const(1, *partition_value); - } + const auto* partition_value = find_partition_value(column, _partition_values); + if (partition_value != nullptr) { + value_column = column.type->create_column_const(1, *partition_value); } block->insert({std::move(value_column), column.type, column.name}); } diff --git a/be/src/format_v2/table_reader.h b/be/src/format_v2/table_reader.h index 48e3d996ce22fd..baf2feb3c4f454 100644 --- a/be/src/format_v2/table_reader.h +++ b/be/src/format_v2/table_reader.h @@ -78,14 +78,10 @@ namespace doris::format { using DeleteRows = std::vector; // Row-level predicates on table/global schema. They are rewritten to file-local expressions when -// possible; otherwise TableReader evaluates them after final table-schema materialization. +// possible, and remain the source of row-level filtering after localization. struct TableFilter { VExprContextSPtr conjunct; std::vector global_indices; - size_t source_conjunct_index = 0; - // False after the first unsafe source conjunct so file-local execution cannot reorder a later - // predicate ahead of stateful or error-preserving table semantics. - bool can_localize = true; }; struct ScanTask { @@ -98,6 +94,7 @@ struct ProjectedColumnBuildContext { const TFileScanRangeParams* scan_params = nullptr; const TFileRangeDesc* range = nullptr; RuntimeState* runtime_state = nullptr; + const SlotDescriptor* slot_desc = nullptr; std::optional schema_column = std::nullopt; size_t next_file_column_idx = 0; }; @@ -116,7 +113,6 @@ struct ReadProfile { RuntimeProfile::Counter* exec_timer = nullptr; RuntimeProfile::Counter* prepare_split_timer = nullptr; RuntimeProfile::Counter* finalize_timer = nullptr; - RuntimeProfile::Counter* residual_filter_timer = nullptr; RuntimeProfile::Counter* create_reader_timer = nullptr; RuntimeProfile::Counter* pushdown_agg_timer = nullptr; RuntimeProfile::Counter* open_reader_timer = nullptr; @@ -133,23 +129,12 @@ struct ReadProfile { RuntimeProfile::Counter* file_reader_close_timer = nullptr; }; -struct MaterializedBlockStats { - bool has_materialized_input = false; - size_t rows = 0; - size_t bytes = 0; - size_t allocated_bytes = 0; -}; - struct TableReadOptions { // Columns need to be read from file and output by table reader. They are all in table/global // schema semantics. const std::vector projected_columns; // All complex conjuncts from scan operator const VExprContextSPtrs conjuncts; - // Number of leading conjuncts whose row-level execution is owned by TableReader/FileReader. - // FileScannerV2 still passes the complete ordered list so mapping, pruning guards, aggregate - // eligibility, and condition-cache analysis see the exact query semantics. nullopt means all. - const std::optional table_reader_owned_conjunct_count = std::nullopt; // File format of the underlying data files, needed for reader initialization and reader-level // filter pushdown. const FileFormat format; @@ -222,10 +207,6 @@ class TableReader { #ifdef BE_TEST size_t TEST_batch_size() const { return _batch_size; } - size_t TEST_conjunct_count() const { return _conjuncts.size(); } - size_t TEST_table_reader_owned_conjunct_count() const { - return _table_reader_owned_conjunct_count; - } void TEST_set_condition_cache_hit_count(int64_t hits) { _condition_cache_hit_count = hits; } bool TEST_current_data_file_is_immutable() const { DORIS_CHECK(_current_task != nullptr); @@ -247,25 +228,6 @@ class TableReader { return _current_split_uses_metadata_count; } - // Runtime filters that arrive after a split has opened cannot be pushed into that file reader. - // Keep their expression contexts in TableReader and evaluate them as residual predicates for - // the active reader; later splits can localize them normally. - virtual Status append_conjuncts(const VExprContextSPtrs& conjuncts); - - // Append a full ordered snapshot delta while marking only its leading prefix as owned by - // TableReader/FileReader. This non-virtual wrapper preserves the long-standing virtual API and - // carries the ownership boundary through hybrid readers to their children. - Status append_conjuncts_with_ownership(const VExprContextSPtrs& conjuncts, - size_t table_reader_owned_conjunct_count); - - // Shared safety classification for deciding which ordered conjunct prefix may execute below - // Scanner without changing stateful or error-preserving semantics. - static bool is_safe_to_pre_execute(const VExprContextSPtr& conjunct); - - virtual const MaterializedBlockStats& last_materialized_block_stats() const { - return _last_materialized_block_stats; - } - // Discard the active split after the caller decides an error is ignorable, for example a // stale external-table file listing that returns NOT_FOUND. The next prepare_split() must start // with no concrete reader or split-local state left from the failed split. @@ -285,7 +247,6 @@ class TableReader { _remaining_file_level_count = -1; _current_split_uses_metadata_count = false; _current_split_pruned = false; - _remaining_conjuncts.clear(); return Status::OK(); } @@ -295,7 +256,6 @@ class TableReader { virtual Status get_block(Block* block, bool* eos) { SCOPED_TIMER(_profile.total_timer); SCOPED_TIMER(_profile.exec_timer); - _last_materialized_block_stats = {}; DORIS_CHECK(block->columns() == _projected_columns.size()); block->clear_column_data(_projected_columns.size()); @@ -368,7 +328,7 @@ class TableReader { RETURN_IF_ERROR(_check_file_block_columns("after file reader get_block", current_rows)); #endif DORIS_CHECK(block->columns() == _data_reader.column_mapper->mappings().size()); - RETURN_IF_ERROR(finalize_chunk(block, ¤t_rows)); + RETURN_IF_ERROR(finalize_chunk(block, current_rows)); #ifndef NDEBUG RETURN_IF_ERROR( _check_table_block_columns("after finalize_chunk", block, current_rows)); @@ -377,13 +337,6 @@ class TableReader { _current_reader_reached_eof = !stopped_during_read; RETURN_IF_ERROR(close_current_reader()); } - if (current_rows == 0) { - // One materialized batch is one Scanner progress unit even when residual - // predicates reject every row. Returning here preserves row-budget and - // cancellation checks in Scanner::get_block(). - block->clear_column_data(_projected_columns.size()); - return Status::OK(); - } return Status::OK(); } } @@ -401,7 +354,6 @@ class TableReader { _remaining_table_level_count = -1; _remaining_file_level_count = -1; _current_split_uses_metadata_count = false; - _remaining_conjuncts.clear(); return Status::OK(); } @@ -487,17 +439,14 @@ class TableReader { // reader with the request. File scan request carries row-level expression filters and // file-level pruning hints. Only expression filters decide returned rows. auto file_request = std::make_shared(); - FilterLocalizationResult localization_result; RETURN_IF_ERROR(_data_reader.column_mapper->create_scan_request( - _table_filters, _projected_columns, file_request.get(), _runtime_state, - &localization_result)); + _table_filters, _projected_columns, file_request.get(), _runtime_state)); bool constant_filter_pruned_split = false; RETURN_IF_ERROR(_evaluate_constant_filters(&constant_filter_pruned_split)); if (constant_filter_pruned_split) { RETURN_IF_ERROR(close_current_reader()); return Status::OK(); } - RETURN_IF_ERROR(_prepare_remaining_conjuncts(localization_result)); // COUNT(*) has no semantic column argument, but Nereids retains a minimum-width scan slot // so the scan node still has an output tuple. Record only the current non-predicate file // columns before table-format hooks add row-position or equality-delete dependencies. This @@ -573,13 +522,9 @@ class TableReader { } Status _build_table_filters_from_conjuncts(); - Status _replace_conjuncts(const VExprContextSPtrs& conjuncts); - Status _prepare_conjunct(const VExprContextSPtr& source, VExprContextSPtr* prepared); - Status _prepare_remaining_conjuncts(const FilterLocalizationResult& localization_result); - Status _prepare_all_conjuncts_as_remaining(); - Status _filter_remaining_conjuncts(Block* block, size_t* rows); Status _evaluate_partition_prune_conjuncts(const VExprContextSPtrs& conjuncts, bool* can_filter_all); + static bool _is_safe_to_pre_execute(const VExprContextSPtr& conjunct); Status _build_partition_prune_block(Block* block) const; Status _open_local_filter_exprs(const FileScanRequest& file_request); Status _init_reader_condition_cache(const FileScanRequest& file_request); @@ -598,7 +543,7 @@ class TableReader { if (table_filter.conjunct == nullptr) { continue; } - DORIS_CHECK(is_safe_to_pre_execute(table_filter.conjunct)); + DORIS_CHECK(_is_safe_to_pre_execute(table_filter.conjunct)); // RuntimeFilterExpr does not implement execute_column_impl(); it is evaluated by the // row-level filter path through execute_filter(). Constant split pruning uses // VExprContext::execute() on a one-row synthetic block, so runtime filters must not be @@ -815,7 +760,6 @@ class TableReader { } _table_filters.clear(); _constant_pruning_safe_filter_count = 0; - _remaining_conjuncts.clear(); _data_reader.file_schema.clear(); _data_reader.file_block_layout.clear(); _data_reader.block_template.clear(); @@ -831,28 +775,15 @@ class TableReader { } } - void _reset_materialized_block_stats() { _last_materialized_block_stats = {}; } - - void _record_materialized_block_stats(const Block& block, size_t rows) { - _last_materialized_block_stats = { - .has_materialized_input = true, - .rows = rows, - .bytes = block.bytes(), - .allocated_bytes = block.allocated_bytes(), - }; - } - // Finalize file-local block to table/global schema block. - Status finalize_chunk(Block* block, size_t* rows) { - DORIS_CHECK(rows != nullptr); + Status finalize_chunk(Block* block, const size_t rows) { SCOPED_TIMER(_profile.finalize_timer); size_t idx = 0; const auto& mappings = _data_reader.column_mapper->mappings(); for (const auto& mapping : mappings) { ColumnPtr column; - RETURN_IF_ERROR(_materialize_mapping_column(mapping, &_data_reader.block_template, - *rows, &column, - idx + 1 == mappings.size())); + RETURN_IF_ERROR(_materialize_mapping_column(mapping, &_data_reader.block_template, rows, + &column, idx + 1 == mappings.size())); block->replace_by_position(idx, IColumn::mutate(std::move(column))); idx++; } @@ -860,12 +791,7 @@ class TableReader { // Enforce CHAR/VARCHAR length declared by the table schema after all file-to-table // materialization has finished. RETURN_IF_ERROR(_truncate_char_or_varchar_columns(block)); - // Preserve the cost of materialization before residual predicates shrink the block. The - // scanner uses this snapshot for bounded progress and adaptive batch sizing. - _record_materialized_block_stats(*block, *rows); - // Predicate ownership is split-local: only predicates not acknowledged as exact by this - // split's FileScanRequest run here, after virtual/default/schema-evolution values exist. - return _filter_remaining_conjuncts(block, rows); + return Status::OK(); } // Materialize virtual columns in the table block, such as Iceberg _row_id and @@ -1029,7 +955,31 @@ class TableReader { // - table VARCHAR(10), file STRING: truncate to 10 because STRING has no declared bound; // - table STRING, any file type: no truncation because the target has no bound. static bool _should_truncate_char_or_varchar_column(const ColumnMapping& mapping) { - return requires_char_or_varchar_truncation(mapping); + if (mapping.table_type == nullptr) { + return false; + } + const auto table_type = remove_nullable(mapping.table_type); + const auto primitive_type = table_type->get_primitive_type(); + if (primitive_type != TYPE_VARCHAR && primitive_type != TYPE_CHAR) { + return false; + } + const auto target_len = assert_cast(table_type.get())->len(); + if (target_len <= 0) { + return false; + } + if (mapping.file_type == nullptr) { + return true; + } + const auto file_type = remove_nullable(mapping.file_type); + DORIS_CHECK(file_type != nullptr); + int file_len = -1; + if (file_type->get_primitive_type() == TYPE_VARCHAR || + file_type->get_primitive_type() == TYPE_CHAR || + file_type->get_primitive_type() == TYPE_STRING) { + file_len = assert_cast(file_type.get())->len(); + } + + return file_len < 0 || target_len < file_len; } // Truncate a materialized CHAR/VARCHAR column in place by reusing the vectorized substring @@ -1126,8 +1076,9 @@ class TableReader { if (!_all_runtime_filters_applied_for_split) { return false; } - // Even a slotless conjunct that cannot become a TableFilter must see every source row - // before an aggregate reduces the stream to synthetic COUNT/MINMAX rows. + // Scanner owns the original conjunct list and evaluates it after TableReader finalizes + // rows. Even a slotless conjunct that cannot become a TableFilter must see every source + // row before an aggregate reduces the stream to synthetic COUNT/MINMAX rows. if (!_conjuncts.empty()) { return false; } @@ -1799,10 +1750,6 @@ class TableReader { // intentionally absent from that vector but must still act as ordering barriers. size_t _constant_pruning_safe_filter_count = 0; VExprContextSPtrs _conjuncts; - size_t _table_reader_owned_conjunct_count = 0; - std::optional _appended_table_reader_owned_conjunct_count; - VExprContextSPtrs _remaining_conjuncts; - MaterializedBlockStats _last_materialized_block_stats; ReadProfile _profile; // Parsed from row-position based delete files, including position delete and deletion vector. DeleteRows* _delete_rows = nullptr; diff --git a/be/src/format_v2/wal/wal_reader.cpp b/be/src/format_v2/wal/wal_reader.cpp new file mode 100644 index 00000000000000..55e88796c9805e --- /dev/null +++ b/be/src/format_v2/wal/wal_reader.cpp @@ -0,0 +1,301 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +#include "format_v2/wal/wal_reader.h" + +#include +#include + +#include +#include +#include + +#include "agent/be_exec_version_manager.h" +#include "common/cast_set.h" +#include "core/block/block.h" +#include "core/data_type/data_type_factory.hpp" +#include "core/data_type/data_type_nullable.h" +#include "format_v2/column_mapper.h" +#include "format_v2/materialized_reader_util.h" +#include "load/group_commit/wal/wal_file_reader.h" +#include "load/group_commit/wal/wal_manager.h" +#include "runtime/exec_env.h" +#include "runtime/runtime_state.h" + +namespace doris::format::wal { +namespace { + +class WalColumnMapper final : public TableColumnMapper { +public: + using TableColumnMapper::TableColumnMapper; + + Status create_mapping(const std::vector& projected_columns, + const std::map& partition_values, + const std::vector& file_schema) override { + for (const auto& projected : projected_columns) { + if (!projected.has_identifier_field_id()) { + return Status::InternalError("WAL projected column {} has no unique id", + projected.name); + } + const auto found = std::ranges::find_if(file_schema, [&](const auto& file_column) { + return file_column.has_identifier_field_id() && + file_column.get_identifier_field_id() == projected.get_identifier_field_id(); + }); + if (found == file_schema.end()) { + return Status::InternalError("WAL does not contain column unique id {} ({})", + projected.get_identifier_field_id(), projected.name); + } + } + return TableColumnMapper::create_mapping(projected_columns, partition_values, file_schema); + } + +protected: + bool enable_lazy_materialization() const override { return false; } + bool force_full_complex_scan_projection() const override { return true; } +}; + +ColumnDefinition build_wal_column_definition(const PColumnMeta& meta, int32_t local_id) { + ColumnDefinition field; + field.local_id = local_id; + field.name = meta.name(); + field.type = make_nullable(DataTypeFactory::instance().create_data_type(meta)); + field.children.reserve(meta.children_size()); + for (int child_idx = 0; child_idx < meta.children_size(); ++child_idx) { + field.children.push_back(build_wal_column_definition(meta.children(child_idx), child_idx)); + } + return field; +} + +} // namespace + +Status parse_wal_column_ids(const std::string& encoded, std::vector* column_ids) { + DORIS_CHECK(column_ids != nullptr); + column_ids->clear(); + if (encoded.empty()) { + return Status::Corruption("WAL header contains no column ids"); + } + + std::unordered_set seen; + for (const absl::string_view token : absl::StrSplit(encoded, ',')) { + int32_t column_id = 0; + if (token.empty() || !absl::SimpleAtoi(token, &column_id)) { + return Status::Corruption("invalid WAL column id '{}'", std::string(token)); + } + if (!seen.emplace(column_id).second) { + return Status::Corruption("duplicate WAL column id {}", column_id); + } + column_ids->push_back(column_id); + } + return Status::OK(); +} + +WalReader::WalReader(std::shared_ptr& system_properties, + std::unique_ptr& file_description, + std::shared_ptr io_ctx, RuntimeProfile* profile, + const std::vector& projected_columns) + : FileReader(system_properties, file_description, std::move(io_ctx), profile), + _projected_columns(projected_columns) {} + +WalReader::~WalReader() { + static_cast(close()); +} + +Status WalReader::init(RuntimeState* state) { + if (state == nullptr || state->exec_env() == nullptr || + state->exec_env()->wal_mgr() == nullptr) { + return Status::InvalidArgument("WAL v2 reader requires a runtime WAL manager"); + } + RETURN_IF_ERROR(state->exec_env()->wal_mgr()->get_wal_path(state->wal_id(), _wal_path)); + _wal_reader = std::make_shared(_wal_path); + RETURN_IF_ERROR(_wal_reader->init()); + + std::string encoded_column_ids; + RETURN_IF_ERROR(_wal_reader->read_header(_version, encoded_column_ids)); + RETURN_IF_ERROR(parse_wal_column_ids(encoded_column_ids, &_column_ids)); + _reader_eof = false; + _eof = false; + return Status::OK(); +} + +Status WalReader::get_schema(std::vector* file_schema) const { + if (file_schema == nullptr) { + return Status::InvalidArgument("WAL v2 file_schema is null"); + } + RETURN_IF_ERROR(_ensure_schema_loaded()); + *file_schema = _file_schema; + return Status::OK(); +} + +std::unique_ptr WalReader::create_column_mapper( + TableColumnMapperOptions options) const { + return std::make_unique(std::move(options)); +} + +Status WalReader::open(std::shared_ptr request) { + RETURN_IF_ERROR(FileReader::open(std::move(request))); + _first_block_consumed = false; + _eof = false; + return Status::OK(); +} + +Status WalReader::get_block(Block* file_block, size_t* rows, bool* eof) { + DORIS_CHECK(file_block != nullptr); + DORIS_CHECK(rows != nullptr); + DORIS_CHECK(eof != nullptr); + if (_request == nullptr) { + return Status::InternalError("WAL v2 reader is not open"); + } + + *rows = 0; + *eof = false; + if (_reader_eof) { + *eof = true; + _eof = true; + return Status::OK(); + } + + PBlock pblock; + if (_first_block_loaded && !_first_block_consumed) { + // Schema discovery owns the first payload temporarily; transfer it into the read path so + // the reader neither retains a second PBlock nor forces protobuf to clone its buffers. + pblock.Swap(&_first_block); + _first_block_consumed = true; + _first_block_loaded = false; + } else { + auto status = _wal_reader->read_block(pblock); + if (status.is()) { + _reader_eof = true; + *eof = true; + _eof = true; + return Status::OK(); + } + RETURN_IF_ERROR(status); + } + RETURN_IF_ERROR(_validate_block_version(pblock)); + + Block source_block; + size_t uncompressed_size = 0; + int64_t decompress_time = 0; + RETURN_IF_ERROR(source_block.deserialize(pblock, &uncompressed_size, &decompress_time)); + if (source_block.columns() != _column_ids.size()) { + return Status::Corruption("WAL block has {} columns but header declares {}", + source_block.columns(), _column_ids.size()); + } + RETURN_IF_ERROR(_materialize_requested_columns(&source_block, file_block)); + *rows = file_block->rows(); + _record_scan_rows(cast_set(*rows)); + RETURN_IF_ERROR( + apply_materialized_reader_filters(_request.get(), _io_ctx.get(), file_block, rows)); + return Status::OK(); +} + +Status WalReader::close() { + _request.reset(); + _reader_eof = true; + _eof = true; + if (_wal_reader == nullptr) { + return Status::OK(); + } + auto status = _wal_reader->finalize(); + if (status.ok()) { + _wal_reader.reset(); + } + return status; +} + +Status WalReader::_ensure_schema_loaded() const { + if (_schema_inited) { + return Status::OK(); + } + + auto status = _wal_reader->read_block(_first_block); + if (status.is()) { + // An empty WAL still has a complete unique-id header. Use only matching projected types; + // there is no data block from which unprojected physical types could be inferred. + return _init_schema_from_block(nullptr); + } + RETURN_IF_ERROR(status); + RETURN_IF_ERROR(_validate_block_version(_first_block)); + _first_block_loaded = true; + return _init_schema_from_block(&_first_block); +} + +Status WalReader::_validate_block_version(const PBlock& pblock) const { + const int version = pblock.has_be_exec_version() ? pblock.be_exec_version() : 0; + if (!BeExecVersionManager::check_be_exec_version(version)) { + return Status::DataQualityError("unsupported BE execution version {} in WAL", version); + } + return Status::OK(); +} + +Status WalReader::_init_schema_from_block(const PBlock* pblock) const { + if (pblock != nullptr && cast_set(pblock->column_metas_size()) != _column_ids.size()) { + return Status::Corruption("WAL block schema has {} columns but header declares {}", + pblock->column_metas_size(), _column_ids.size()); + } + + _file_schema.clear(); + for (size_t idx = 0; idx < _column_ids.size(); ++idx) { + ColumnDefinition field; + field.identifier = Field::create_field(_column_ids[idx]); + field.local_id = cast_set(idx); + if (pblock != nullptr) { + const auto& meta = pblock->column_metas(cast_set(idx)); + // WAL metadata is the file-local schema; preserve its complete nested shape so the + // mapper can validate ARRAY/MAP/STRUCT projections instead of seeing an empty shell. + field = build_wal_column_definition(meta, cast_set(idx)); + field.identifier = Field::create_field(_column_ids[idx]); + } else { + const auto projected = + std::ranges::find_if(_projected_columns, [&](const auto& candidate) { + return candidate.has_identifier_field_id() && + candidate.get_identifier_field_id() == _column_ids[idx]; + }); + if (projected == _projected_columns.end()) { + continue; + } + field.name = projected->name; + field.type = projected->type; + } + _file_schema.push_back(std::move(field)); + } + _schema_inited = true; + return Status::OK(); +} + +Status WalReader::_materialize_requested_columns(Block* source_block, Block* file_block) const { + DORIS_CHECK(source_block != nullptr); + for (const auto& [file_column_id, block_position] : _request->local_positions) { + const auto source_idx = file_column_id.value(); + if (source_idx < 0 || cast_set(source_idx) >= source_block->columns()) { + return Status::Corruption("WAL request refers to invalid local column {}", source_idx); + } + if (block_position.value() >= file_block->columns()) { + return Status::InternalError("WAL request has invalid block position {}", + block_position.value()); + } + const auto& target = file_block->get_by_position(block_position.value()); + // Deserialized WAL columns have a single owner. Move that ownership into the output block + // before mutate() so wide payloads do not trigger copy-on-write deep clones. + auto column = std::move(source_block->get_by_position(source_idx).column); + column = make_column_nullable_if_needed(std::move(column), target.type); + file_block->replace_by_position(block_position.value(), IColumn::mutate(std::move(column))); + } + return Status::OK(); +} + +} // namespace doris::format::wal diff --git a/be/src/format_v2/wal/wal_reader.h b/be/src/format_v2/wal/wal_reader.h new file mode 100644 index 00000000000000..894822352fd7c7 --- /dev/null +++ b/be/src/format_v2/wal/wal_reader.h @@ -0,0 +1,72 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +#pragma once + +#include + +#include +#include +#include +#include + +#include "format_v2/file_reader.h" + +namespace doris { +class WalFileReader; +} + +namespace doris::format::wal { + +Status parse_wal_column_ids(const std::string& encoded, std::vector* column_ids); + +class WalReader final : public FileReader { +public: + WalReader(std::shared_ptr& system_properties, + std::unique_ptr& file_description, + std::shared_ptr io_ctx, RuntimeProfile* profile, + const std::vector& projected_columns); + ~WalReader() override; + + Status init(RuntimeState* state) override; + Status get_schema(std::vector* file_schema) const override; + std::unique_ptr create_column_mapper( + TableColumnMapperOptions options) const override; + Status open(std::shared_ptr request) override; + Status get_block(Block* file_block, size_t* rows, bool* eof) override; + Status close() override; + +private: + Status _ensure_schema_loaded() const; + Status _validate_block_version(const PBlock& pblock) const; + Status _init_schema_from_block(const PBlock* pblock) const; + Status _materialize_requested_columns(Block* source_block, Block* file_block) const; + + const std::vector _projected_columns; + std::shared_ptr _wal_reader; + std::string _wal_path; + uint32_t _version = 0; + std::vector _column_ids; + mutable std::vector _file_schema; + mutable PBlock _first_block; + mutable bool _first_block_loaded = false; + mutable bool _first_block_consumed = false; + mutable bool _schema_inited = false; + bool _reader_eof = false; +}; + +} // namespace doris::format::wal diff --git a/be/src/format_v2/wal/wal_table_reader.cpp b/be/src/format_v2/wal/wal_table_reader.cpp new file mode 100644 index 00000000000000..428ae7709af790 --- /dev/null +++ b/be/src/format_v2/wal/wal_table_reader.cpp @@ -0,0 +1,47 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +#include "format_v2/wal/wal_table_reader.h" + +#include "format_v2/wal/wal_reader.h" +#include "runtime/descriptors.h" + +namespace doris::format::wal { + +Status WalTableReader::annotate_projected_column(const TFileScanSlotInfo&, + ProjectedColumnBuildContext* context, + ColumnDefinition* column) const { + DORIS_CHECK(context != nullptr); + DORIS_CHECK(column != nullptr); + if (context->slot_desc == nullptr || context->slot_desc->col_unique_id() < 0) { + return Status::InternalError("WAL projected column {} has no valid unique id", + column->name); + } + // WAL headers carry stable Doris column unique ids, so name-based matching would return a + // renamed column from the wrong physical position. + column->identifier = Field::create_field(context->slot_desc->col_unique_id()); + return Status::OK(); +} + +Status WalTableReader::create_file_reader(std::unique_ptr* reader) { + DORIS_CHECK(reader != nullptr); + *reader = std::make_unique(_system_properties, _current_task->data_file, _io_ctx, + _scanner_profile, _projected_columns); + return Status::OK(); +} + +} // namespace doris::format::wal diff --git a/be/src/format_v2/wal/wal_table_reader.h b/be/src/format_v2/wal/wal_table_reader.h new file mode 100644 index 00000000000000..b172c21f87f451 --- /dev/null +++ b/be/src/format_v2/wal/wal_table_reader.h @@ -0,0 +1,37 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +#pragma once + +#include "format_v2/table_reader.h" + +namespace doris::format::wal { + +class WalTableReader final : public TableReader { +public: + Status annotate_projected_column(const TFileScanSlotInfo& slot_info, + ProjectedColumnBuildContext* context, + ColumnDefinition* column) const override; + +protected: + Status create_file_reader(std::unique_ptr* reader) override; + TableColumnMappingMode mapping_mode() const override { + return TableColumnMappingMode::BY_FIELD_ID; + } +}; + +} // namespace doris::format::wal diff --git a/be/src/runtime/query_context.h b/be/src/runtime/query_context.h index 5e2b6babe87305..3ef113c1fd1097 100644 --- a/be/src/runtime/query_context.h +++ b/be/src/runtime/query_context.h @@ -190,12 +190,6 @@ class QueryContext : public std::enable_shared_from_this { return _query_options.__isset.enable_force_spill && _query_options.enable_force_spill; } const TQueryOptions& query_options() const { return _query_options; } - bool should_be_shuffled_agg(int node_id) const { - return _query_options.__isset.shuffled_agg_ids && - std::any_of(_query_options.shuffled_agg_ids.begin(), - _query_options.shuffled_agg_ids.end(), - [&](const int id) -> bool { return id == node_id; }); - } // global runtime filter mgr, the runtime filter have remote target or // need local merge should regist here. before publish() or push_to_remote() diff --git a/be/src/runtime/runtime_profile.cpp b/be/src/runtime/runtime_profile.cpp index 15e5b472d90b8a..8c7e9170758c63 100644 --- a/be/src/runtime/runtime_profile.cpp +++ b/be/src/runtime/runtime_profile.cpp @@ -394,6 +394,23 @@ RuntimeProfile* RuntimeProfile::create_child(const std::string& name, bool inden return child; } +RuntimeProfile* RuntimeProfile::get_or_create_child(const std::string& name, bool indent, + bool prepend) { + std::lock_guard l(_children_lock); + auto it = _child_map.find(name); + if (it != _child_map.end()) { + return it->second; + } + + RuntimeProfile* child = _pool->add(new RuntimeProfile(name)); + if (this->is_set_metadata()) { + child->set_metadata(this->metadata()); + } + auto* location = !_children.empty() && prepend ? _children.front().first : nullptr; + add_child_unlock(child, indent, location); + return child; +} + void RuntimeProfile::add_child_unlock(RuntimeProfile* child, bool indent, RuntimeProfile* loc) { DCHECK(child != nullptr); _child_map[child->_name] = child; diff --git a/be/src/runtime/runtime_profile.h b/be/src/runtime/runtime_profile.h index 32f26bf77fbfa8..d7948a5a3f9c8d 100644 --- a/be/src/runtime/runtime_profile.h +++ b/be/src/runtime/runtime_profile.h @@ -499,6 +499,11 @@ class RuntimeProfile { /// otherwise appended after other child profiles. RuntimeProfile* create_child(const std::string& name, bool indent = true, bool prepend = false); + /// Returns an existing child profile with 'name', or creates it if absent. Lookup and creation + /// are atomic so concurrent callers cannot race while initializing a shared profile subtree. + RuntimeProfile* get_or_create_child(const std::string& name, bool indent = true, + bool prepend = false); + // Merges the src profile into this one, combining counters that have an identical // path. Info strings from profiles are not merged. 'src' would be a const if it // weren't for locking. diff --git a/be/src/runtime/runtime_state.h b/be/src/runtime/runtime_state.h index 40ba041ffab388..fc764085c8698a 100644 --- a/be/src/runtime/runtime_state.h +++ b/be/src/runtime/runtime_state.h @@ -585,6 +585,11 @@ class RuntimeState { _query_options.enable_streaming_agg_hash_join_force_passthrough; } + bool enable_local_exchange_before_agg() const { + return _query_options.__isset.enable_local_exchange_before_agg && + _query_options.enable_local_exchange_before_agg; + } + bool enable_distinct_streaming_agg_force_passthrough() const { return _query_options.__isset.enable_distinct_streaming_agg_force_passthrough && _query_options.enable_distinct_streaming_agg_force_passthrough; diff --git a/be/src/storage/segment/adaptive_block_size_predictor.cpp b/be/src/storage/segment/adaptive_block_size_predictor.cpp index 7a5ad573a2ea6c..d8cc700f579853 100644 --- a/be/src/storage/segment/adaptive_block_size_predictor.cpp +++ b/be/src/storage/segment/adaptive_block_size_predictor.cpp @@ -32,14 +32,11 @@ AdaptiveBlockSizePredictor::AdaptiveBlockSizePredictor(size_t preferred_block_si _metadata_hint_bytes_per_row(metadata_hint_bytes_per_row) {} void AdaptiveBlockSizePredictor::update(const Block& block) { - update(block.rows(), block.bytes()); -} - -void AdaptiveBlockSizePredictor::update(size_t rows, size_t bytes) { + size_t rows = block.rows(); if (rows == 0) { return; } - double cur = static_cast(bytes) / static_cast(rows); + double cur = static_cast(block.bytes()) / static_cast(rows); if (!_has_history) { _bytes_per_row = cur; diff --git a/be/src/storage/segment/adaptive_block_size_predictor.h b/be/src/storage/segment/adaptive_block_size_predictor.h index f327fec8517f63..e03f18c2a536d2 100644 --- a/be/src/storage/segment/adaptive_block_size_predictor.h +++ b/be/src/storage/segment/adaptive_block_size_predictor.h @@ -60,7 +60,6 @@ class AdaptiveBlockSizePredictor { // Update EWMA estimates from a completed batch. Must be called only when block.rows() > 0 // and the batch returned Status::OK(). void update(const Block& block); - void update(size_t rows, size_t bytes); // Predict how many rows the next batch should read. // Never exceeds |block_size_rows|; never returns less than 1. diff --git a/be/src/util/simd/parquet_kernels.cpp b/be/src/util/simd/parquet_kernels.cpp new file mode 100644 index 00000000000000..9514ac445b09c3 --- /dev/null +++ b/be/src/util/simd/parquet_kernels.cpp @@ -0,0 +1,651 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +#include "util/simd/parquet_kernels.h" + +#include +#include +#include +#include +#include + +#if defined(__x86_64__) && (defined(__GNUC__) || defined(__clang__)) +#include +#define DORIS_PARQUET_X86_SIMD +#endif + +namespace doris::simd { +namespace { + +#ifdef DORIS_PARQUET_X86_SIMD +// Keep x86-only dispatch helpers out of scalar-only builds so warning-clean ARM builds do not +// depend on compiler-specific unused-function behavior. +bool has_avx2() { + return __builtin_cpu_supports("avx2"); +} + +void byte_stream_split_decode_scalar(const uint8_t* src, size_t width, size_t offset, + size_t num_values, size_t stride, uint8_t* dest) { + for (size_t row = 0; row < num_values; ++row) { + for (size_t byte = 0; byte < width; ++byte) { + dest[row * width + byte] = src[byte * stride + offset + row]; + } + } +} +#endif + +template +bool scalar_compare(T lhs, T rhs, RawComparisonOp op) { + const auto equal = [](T left, T right) { + if constexpr (std::is_floating_point_v) { + return (std::isnan(left) && std::isnan(right)) || left == right; + } + return left == right; + }; + const auto greater = [](T left, T right) { + if constexpr (std::is_floating_point_v) { + // Match Doris Compare: NaN is equal to NaN and greater than every finite value. + if (std::isnan(right)) { + return false; + } + if (std::isnan(left)) { + return true; + } + } + return left > right; + }; + switch (op) { + case RawComparisonOp::EQ: + return equal(lhs, rhs); + case RawComparisonOp::NE: + return !equal(lhs, rhs); + case RawComparisonOp::LT: + return greater(rhs, lhs); + case RawComparisonOp::LE: + return !greater(lhs, rhs); + case RawComparisonOp::GT: + return greater(lhs, rhs); + case RawComparisonOp::GE: + return !greater(rhs, lhs); + } + __builtin_unreachable(); +} + +#ifdef DORIS_PARQUET_X86_SIMD +template +__attribute__((target("avx2"))) void byte_stream_split_decode_avx2(const uint8_t* src, + size_t offset, size_t num_values, + size_t stride, uint8_t* dest) { + static_assert(WIDTH == 4 || WIDTH == 8); + constexpr size_t STEPS = WIDTH == 8 ? 3 : 2; + constexpr size_t LANES = sizeof(__m256i); + const size_t blocks = num_values / LANES; + __m256i stage[STEPS + 1][WIDTH]; + __m256i result[WIDTH]; + + for (size_t block = 0; block < blocks; ++block) { + for (size_t stream = 0; stream < WIDTH; ++stream) { + stage[0][stream] = _mm256_loadu_si256(reinterpret_cast( + src + stream * stride + offset + block * LANES)); + } + for (size_t step = 0; step < STEPS; ++step) { + // AVX2 unpack instructions operate independently in each 128-bit half. Keep the + // byte-transpose hierarchy lane-local, then stitch the halves only after the last + // unpack so every output vector contains contiguous decoded rows. + for (size_t pair = 0; pair < WIDTH / 2; ++pair) { + stage[step + 1][pair * 2] = + _mm256_unpacklo_epi8(stage[step][pair], stage[step][WIDTH / 2 + pair]); + stage[step + 1][pair * 2 + 1] = + _mm256_unpackhi_epi8(stage[step][pair], stage[step][WIDTH / 2 + pair]); + } + } + for (size_t pair = 0; pair < WIDTH / 2; ++pair) { + result[pair] = _mm256_permute2x128_si256(stage[STEPS][pair * 2], + stage[STEPS][pair * 2 + 1], 0x20); + result[WIDTH / 2 + pair] = _mm256_permute2x128_si256(stage[STEPS][pair * 2], + stage[STEPS][pair * 2 + 1], 0x31); + } + for (size_t lane = 0; lane < WIDTH; ++lane) { + _mm256_storeu_si256(reinterpret_cast<__m256i*>(dest + (block * WIDTH + lane) * LANES), + result[lane]); + } + } + const size_t processed = blocks * LANES; + byte_stream_split_decode_scalar(src, WIDTH, offset + processed, num_values - processed, stride, + dest + processed * WIDTH); +} + +__attribute__((target("avx2"))) void delta_decode_int32_avx2(int32_t* values, size_t count, + int32_t min_delta, + int32_t* last_value) { + const __m256i min_delta_vec = _mm256_set1_epi32(min_delta); + const size_t vector_count = count / 8 * 8; + for (size_t row = 0; row < vector_count; row += 8) { + __m256i value = _mm256_loadu_si256(reinterpret_cast(values + row)); + value = _mm256_add_epi32(value, min_delta_vec); + value = _mm256_add_epi32(value, _mm256_slli_si256(value, 4)); + value = _mm256_add_epi32(value, _mm256_slli_si256(value, 8)); + _mm256_storeu_si256(reinterpret_cast<__m256i*>(values + row), value); + } + __m128i carry = _mm_set1_epi32(*last_value); + for (size_t row = 0; row < vector_count; row += 4) { + __m128i value = _mm_loadu_si128(reinterpret_cast(values + row)); + value = _mm_add_epi32(value, carry); + _mm_storeu_si128(reinterpret_cast<__m128i*>(values + row), value); + carry = _mm_shuffle_epi32(value, _MM_SHUFFLE(3, 3, 3, 3)); + } + if (vector_count != 0) { + *last_value = _mm_cvtsi128_si32(carry); + } + using Unsigned = uint32_t; + for (size_t row = vector_count; row < count; ++row) { + values[row] = static_cast(static_cast(values[row]) + + static_cast(min_delta) + + static_cast(*last_value)); + *last_value = values[row]; + } +} + +__attribute__((target("avx2"))) void delta_decode_int64_avx2(int64_t* values, size_t count, + int64_t min_delta, + int64_t* last_value) { + const __m256i min_delta_vec = _mm256_set1_epi64x(min_delta); + const size_t vector_count = count / 4 * 4; + for (size_t row = 0; row < vector_count; row += 4) { + __m256i value = _mm256_loadu_si256(reinterpret_cast(values + row)); + value = _mm256_add_epi64(value, min_delta_vec); + value = _mm256_add_epi64(value, _mm256_slli_si256(value, 8)); + _mm256_storeu_si256(reinterpret_cast<__m256i*>(values + row), value); + } + __m128i carry = _mm_set1_epi64x(*last_value); + for (size_t row = 0; row < vector_count; row += 2) { + __m128i value = _mm_loadu_si128(reinterpret_cast(values + row)); + value = _mm_add_epi64(value, carry); + _mm_storeu_si128(reinterpret_cast<__m128i*>(values + row), value); + carry = _mm_unpackhi_epi64(value, value); + } + if (vector_count != 0) { + *last_value = _mm_cvtsi128_si64(carry); + } + using Unsigned = uint64_t; + for (size_t row = vector_count; row < count; ++row) { + values[row] = static_cast(static_cast(values[row]) + + static_cast(min_delta) + + static_cast(*last_value)); + *last_value = values[row]; + } +} + +__attribute__((target("avx2"))) void dictionary_gather_avx2(const uint8_t* dictionary, + const uint32_t* indices, size_t count, + size_t value_width, uint8_t* dest) { + size_t row = 0; + if (value_width == 4) { + for (; row + 8 <= count; row += 8) { + const __m256i ids = _mm256_loadu_si256(reinterpret_cast(indices + row)); + const __m256i gathered = + _mm256_i32gather_epi32(reinterpret_cast(dictionary), ids, 4); + _mm256_storeu_si256(reinterpret_cast<__m256i*>(dest + row * 4), gathered); + } + } else { + for (; row + 4 <= count; row += 4) { + const __m128i ids = _mm_loadu_si128(reinterpret_cast(indices + row)); + const __m256i gathered = + _mm256_i32gather_epi64(reinterpret_cast(dictionary), ids, 8); + _mm256_storeu_si256(reinterpret_cast<__m256i*>(dest + row * 8), gathered); + } + } + for (; row < count; ++row) { + memcpy(dest + row * value_width, dictionary + indices[row] * value_width, value_width); + } +} + +template +constexpr auto make_expand_permute_lut() { + std::array, 1U << LANES> lut {}; + for (size_t mask = 0; mask < lut.size(); ++mask) { + int32_t source = 0; + for (size_t lane = 0; lane < LANES; ++lane) { + const int32_t value = (mask & (1U << lane)) != 0 ? source++ : 0; + if constexpr (LANES == 8) { + lut[mask][lane] = value; + } else { + lut[mask][lane * 2] = value * 2; + lut[mask][lane * 2 + 1] = value * 2 + 1; + } + } + } + return lut; +} + +constexpr auto EXPAND_PERMUTE_32 = make_expand_permute_lut<8>(); +constexpr auto EXPAND_PERMUTE_64 = make_expand_permute_lut<4>(); + +__attribute__((target("avx2"))) void expand_nullable_avx2(uint8_t* bytes, size_t compact_count, + const uint8_t* nulls, size_t output_count, + size_t value_width) { + size_t source = compact_count; + size_t output = output_count; + if (value_width == 4) { + auto* values = reinterpret_cast(bytes); + while (output >= 8) { + const size_t start = output - 8; + uint32_t valid_mask = 0; + for (size_t lane = 0; lane < 8; ++lane) { + valid_mask |= static_cast(nulls[start + lane] == 0) << lane; + } + const size_t valid = std::popcount(valid_mask); + source -= valid; + const __m256i load_mask = _mm256_cmpgt_epi32(_mm256_set1_epi32(static_cast(valid)), + _mm256_setr_epi32(0, 1, 2, 3, 4, 5, 6, 7)); + const __m256i compact = _mm256_maskload_epi32(values + source, load_mask); + const __m256i permute = _mm256_loadu_si256( + reinterpret_cast(EXPAND_PERMUTE_32[valid_mask].data())); + __m256i expanded = _mm256_permutevar8x32_epi32(compact, permute); + const __m256i valid_lanes = _mm256_setr_epi32( + -(valid_mask & 1U), -((valid_mask >> 1) & 1U), -((valid_mask >> 2) & 1U), + -((valid_mask >> 3) & 1U), -((valid_mask >> 4) & 1U), -((valid_mask >> 5) & 1U), + -((valid_mask >> 6) & 1U), -((valid_mask >> 7) & 1U)); + expanded = _mm256_and_si256(expanded, valid_lanes); + _mm256_storeu_si256(reinterpret_cast<__m256i*>(values + start), expanded); + output = start; + } + } else { + auto* values = reinterpret_cast(bytes); + while (output >= 4) { + const size_t start = output - 4; + uint32_t valid_mask = 0; + for (size_t lane = 0; lane < 4; ++lane) { + valid_mask |= static_cast(nulls[start + lane] == 0) << lane; + } + const size_t valid = std::popcount(valid_mask); + source -= valid; + const __m256i load_mask = + _mm256_setr_epi64x(valid > 0 ? -1LL : 0, valid > 1 ? -1LL : 0, + valid > 2 ? -1LL : 0, valid > 3 ? -1LL : 0); + const __m256i compact = _mm256_maskload_epi64( + reinterpret_cast(values + source), load_mask); + const __m256i permute = _mm256_loadu_si256( + reinterpret_cast(EXPAND_PERMUTE_64[valid_mask].data())); + __m256i expanded = _mm256_permutevar8x32_epi32(compact, permute); + const __m256i valid_lanes = _mm256_setr_epi64x( + (valid_mask & 1U) != 0 ? -1LL : 0, (valid_mask & 2U) != 0 ? -1LL : 0, + (valid_mask & 4U) != 0 ? -1LL : 0, (valid_mask & 8U) != 0 ? -1LL : 0); + expanded = _mm256_and_si256(expanded, valid_lanes); + _mm256_storeu_si256(reinterpret_cast<__m256i*>(values + start), expanded); + output = start; + } + } + while (output > 0) { + --output; + if (nulls[output] != 0) { + memset(bytes + output * value_width, 0, value_width); + } else { + --source; + memmove(bytes + output * value_width, bytes + source * value_width, value_width); + } + } +} + +__attribute__((target("avx2"))) __m256i vector_all(__m256i) { + return _mm256_set1_epi32(-1); +} + +__attribute__((target("avx2"))) __m256 vector_all(__m256) { + return _mm256_castsi256_ps(_mm256_set1_epi32(-1)); +} + +__attribute__((target("avx2"))) __m256d vector_all(__m256d) { + return _mm256_castsi256_pd(_mm256_set1_epi32(-1)); +} + +__attribute__((target("avx2"))) __m256i vector_or(__m256i lhs, __m256i rhs) { + return _mm256_or_si256(lhs, rhs); +} + +__attribute__((target("avx2"))) __m256 vector_or(__m256 lhs, __m256 rhs) { + return _mm256_or_ps(lhs, rhs); +} + +__attribute__((target("avx2"))) __m256d vector_or(__m256d lhs, __m256d rhs) { + return _mm256_or_pd(lhs, rhs); +} + +__attribute__((target("avx2"))) __m256i vector_xor(__m256i lhs, __m256i rhs) { + return _mm256_xor_si256(lhs, rhs); +} + +__attribute__((target("avx2"))) __m256 vector_xor(__m256 lhs, __m256 rhs) { + return _mm256_xor_ps(lhs, rhs); +} + +__attribute__((target("avx2"))) __m256d vector_xor(__m256d lhs, __m256d rhs) { + return _mm256_xor_pd(lhs, rhs); +} + +template +__attribute__((target("avx2"))) Vec combine_comparison(Vec equal, Vec greater, Vec less, + RawComparisonOp op) { + // Target features do not propagate from AVX2 callers into separately instantiated helpers. + // Keep the complete vector operation inside its own target scope for baseline x86 builds. + const Vec all = vector_all(equal); + switch (op) { + case RawComparisonOp::EQ: + return equal; + case RawComparisonOp::NE: + return vector_xor(equal, all); + case RawComparisonOp::LT: + return less; + case RawComparisonOp::LE: + return vector_or(less, equal); + case RawComparisonOp::GT: + return greater; + case RawComparisonOp::GE: + return vector_or(greater, equal); + } + __builtin_unreachable(); +} + +__attribute__((target("avx2"))) void raw_compare_int32_avx2(const uint8_t* bytes, size_t count, + int32_t literal, RawComparisonOp op, + uint8_t* matches) { + size_t row = 0; + const __m256i rhs = _mm256_set1_epi32(literal); + for (; row + 8 <= count; row += 8) { + const __m256i lhs = _mm256_loadu_si256(reinterpret_cast(bytes + row * 4)); + const __m256i equal = _mm256_cmpeq_epi32(lhs, rhs); + const __m256i greater = _mm256_cmpgt_epi32(lhs, rhs); + const __m256i less = _mm256_cmpgt_epi32(rhs, lhs); + const uint32_t mask = static_cast(_mm256_movemask_ps( + _mm256_castsi256_ps(combine_comparison(equal, greater, less, op)))); + for (size_t lane = 0; lane < 8; ++lane) { + matches[row + lane] &= static_cast((mask >> lane) & 1U); + } + } + for (; row < count; ++row) { + int32_t value; + memcpy(&value, bytes + row * 4, sizeof(value)); + bool keep = false; + switch (op) { + case RawComparisonOp::EQ: + keep = value == literal; + break; + case RawComparisonOp::NE: + keep = value != literal; + break; + case RawComparisonOp::LT: + keep = value < literal; + break; + case RawComparisonOp::LE: + keep = value <= literal; + break; + case RawComparisonOp::GT: + keep = value > literal; + break; + case RawComparisonOp::GE: + keep = value >= literal; + break; + } + matches[row] &= static_cast(keep); + } +} + +__attribute__((target("avx2"))) void raw_compare_int64_avx2(const uint8_t* bytes, size_t count, + int64_t literal, RawComparisonOp op, + uint8_t* matches) { + size_t row = 0; + const __m256i rhs = _mm256_set1_epi64x(literal); + for (; row + 4 <= count; row += 4) { + const __m256i lhs = _mm256_loadu_si256(reinterpret_cast(bytes + row * 8)); + const __m256i equal = _mm256_cmpeq_epi64(lhs, rhs); + const __m256i greater = _mm256_cmpgt_epi64(lhs, rhs); + const __m256i less = _mm256_cmpgt_epi64(rhs, lhs); + const uint32_t mask = static_cast(_mm256_movemask_pd( + _mm256_castsi256_pd(combine_comparison(equal, greater, less, op)))); + for (size_t lane = 0; lane < 4; ++lane) { + matches[row + lane] &= static_cast((mask >> lane) & 1U); + } + } + for (; row < count; ++row) { + int64_t value; + memcpy(&value, bytes + row * 8, sizeof(value)); + bool keep = false; + switch (op) { + case RawComparisonOp::EQ: + keep = value == literal; + break; + case RawComparisonOp::NE: + keep = value != literal; + break; + case RawComparisonOp::LT: + keep = value < literal; + break; + case RawComparisonOp::LE: + keep = value <= literal; + break; + case RawComparisonOp::GT: + keep = value > literal; + break; + case RawComparisonOp::GE: + keep = value >= literal; + break; + } + matches[row] &= static_cast(keep); + } +} + +__attribute__((target("avx2"))) void raw_compare_float_avx2(const uint8_t* bytes, size_t count, + float literal, RawComparisonOp op, + uint8_t* matches) { + size_t row = 0; + const __m256 rhs = _mm256_set1_ps(literal); + for (; row + 8 <= count; row += 8) { + const __m256 lhs = _mm256_loadu_ps(reinterpret_cast(bytes + row * 4)); + const __m256 lhs_nan = _mm256_cmp_ps(lhs, lhs, _CMP_UNORD_Q); + const __m256 rhs_nan = _mm256_cmp_ps(rhs, rhs, _CMP_UNORD_Q); + const __m256 both_nan = _mm256_and_ps(lhs_nan, rhs_nan); + const __m256 equal = _mm256_or_ps(_mm256_cmp_ps(lhs, rhs, _CMP_EQ_OQ), both_nan); + const __m256 greater = _mm256_or_ps(_mm256_cmp_ps(lhs, rhs, _CMP_GT_OQ), + _mm256_andnot_ps(rhs_nan, lhs_nan)); + const __m256 less = _mm256_or_ps(_mm256_cmp_ps(lhs, rhs, _CMP_LT_OQ), + _mm256_andnot_ps(lhs_nan, rhs_nan)); + const uint32_t mask = static_cast( + _mm256_movemask_ps(combine_comparison(equal, greater, less, op))); + for (size_t lane = 0; lane < 8; ++lane) { + matches[row + lane] &= static_cast((mask >> lane) & 1U); + } + } + for (; row < count; ++row) { + float value; + memcpy(&value, bytes + row * 4, sizeof(value)); + matches[row] &= static_cast(scalar_compare(value, literal, op)); + } +} + +__attribute__((target("avx2"))) void raw_compare_double_avx2(const uint8_t* bytes, size_t count, + double literal, RawComparisonOp op, + uint8_t* matches) { + size_t row = 0; + const __m256d rhs = _mm256_set1_pd(literal); + for (; row + 4 <= count; row += 4) { + const __m256d lhs = _mm256_loadu_pd(reinterpret_cast(bytes + row * 8)); + const __m256d lhs_nan = _mm256_cmp_pd(lhs, lhs, _CMP_UNORD_Q); + const __m256d rhs_nan = _mm256_cmp_pd(rhs, rhs, _CMP_UNORD_Q); + const __m256d both_nan = _mm256_and_pd(lhs_nan, rhs_nan); + const __m256d equal = _mm256_or_pd(_mm256_cmp_pd(lhs, rhs, _CMP_EQ_OQ), both_nan); + const __m256d greater = _mm256_or_pd(_mm256_cmp_pd(lhs, rhs, _CMP_GT_OQ), + _mm256_andnot_pd(rhs_nan, lhs_nan)); + const __m256d less = _mm256_or_pd(_mm256_cmp_pd(lhs, rhs, _CMP_LT_OQ), + _mm256_andnot_pd(lhs_nan, rhs_nan)); + const uint32_t mask = static_cast( + _mm256_movemask_pd(combine_comparison(equal, greater, less, op))); + for (size_t lane = 0; lane < 4; ++lane) { + matches[row + lane] &= static_cast((mask >> lane) & 1U); + } + } + for (; row < count; ++row) { + double value; + memcpy(&value, bytes + row * 8, sizeof(value)); + matches[row] &= static_cast(scalar_compare(value, literal, op)); + } +} +#endif + +template +void delta_decode_scalar(T* values, size_t count, T min_delta, T* last_value) { + using Unsigned = std::make_unsigned_t; + for (size_t row = 0; row < count; ++row) { + values[row] = static_cast(static_cast(values[row]) + + static_cast(min_delta) + + static_cast(*last_value)); + *last_value = values[row]; + } +} + +template +void raw_compare_scalar(const uint8_t* bytes, size_t count, T literal, RawComparisonOp op, + uint8_t* matches) { + for (size_t row = 0; row < count; ++row) { + if (matches[row] == 0) { + continue; + } + T value; + memcpy(&value, bytes + row * sizeof(T), sizeof(T)); + matches[row] = static_cast(scalar_compare(value, literal, op)); + } +} + +} // namespace + +bool try_byte_stream_split_decode(const uint8_t* src, size_t width, size_t offset, + size_t num_values, size_t stride, uint8_t* dest) { +#ifdef DORIS_PARQUET_X86_SIMD + if (has_avx2() && num_values >= 32 && (width == 4 || width == 8)) { + if (width == 4) { + byte_stream_split_decode_avx2<4>(src, offset, num_values, stride, dest); + } else { + byte_stream_split_decode_avx2<8>(src, offset, num_values, stride, dest); + } + return true; + } +#endif + return false; +} + +void delta_decode(int32_t* values, size_t count, int32_t min_delta, int32_t* last_value) { +#ifdef DORIS_PARQUET_X86_SIMD + if (has_avx2() && count >= 8) { + delta_decode_int32_avx2(values, count, min_delta, last_value); + return; + } +#endif + delta_decode_scalar(values, count, min_delta, last_value); +} + +void delta_decode(int64_t* values, size_t count, int64_t min_delta, int64_t* last_value) { +#ifdef DORIS_PARQUET_X86_SIMD + if (has_avx2() && count >= 4) { + delta_decode_int64_avx2(values, count, min_delta, last_value); + return; + } +#endif + delta_decode_scalar(values, count, min_delta, last_value); +} + +void dictionary_gather(const uint8_t* dictionary, const uint32_t* indices, size_t count, + size_t value_width, uint8_t* dest) { +#ifdef DORIS_PARQUET_X86_SIMD + if (has_avx2() && ((value_width == 4 && count >= 8) || (value_width == 8 && count >= 4))) { + dictionary_gather_avx2(dictionary, indices, count, value_width, dest); + return; + } +#endif + for (size_t row = 0; row < count; ++row) { + memcpy(dest + row * value_width, dictionary + indices[row] * value_width, value_width); + } +} + +void expand_nullable_values(uint8_t* values, size_t compact_count, const uint8_t* nulls, + size_t output_count, size_t value_width) { +#ifdef DORIS_PARQUET_X86_SIMD + if (has_avx2() && + ((value_width == 4 && output_count >= 8) || (value_width == 8 && output_count >= 4))) { + // Backward expansion is required because the compact input and expanded output alias. + // Each SIMD block loads all of its source lanes before overwriting the wider destination. + expand_nullable_avx2(values, compact_count, nulls, output_count, value_width); + return; + } +#endif + size_t source = compact_count; + for (size_t output = output_count; output > 0;) { + --output; + if (nulls[output] != 0) { + memset(values + output * value_width, 0, value_width); + } else { + --source; + memmove(values + output * value_width, values + source * value_width, value_width); + } + } +} + +void raw_compare(const uint8_t* values, size_t count, int32_t literal, RawComparisonOp op, + uint8_t* matches) { +#ifdef DORIS_PARQUET_X86_SIMD + if (has_avx2() && count >= 8) { + raw_compare_int32_avx2(values, count, literal, op, matches); + return; + } +#endif + raw_compare_scalar(values, count, literal, op, matches); +} + +void raw_compare(const uint8_t* values, size_t count, int64_t literal, RawComparisonOp op, + uint8_t* matches) { +#ifdef DORIS_PARQUET_X86_SIMD + if (has_avx2() && count >= 4) { + raw_compare_int64_avx2(values, count, literal, op, matches); + return; + } +#endif + raw_compare_scalar(values, count, literal, op, matches); +} + +void raw_compare(const uint8_t* values, size_t count, float literal, RawComparisonOp op, + uint8_t* matches) { +#ifdef DORIS_PARQUET_X86_SIMD + if (has_avx2() && count >= 8) { + // Doris orders NaN above every finite value and considers NaN equal to NaN. The SIMD + // masks deliberately reconstruct that total order instead of using ordered FP compares. + raw_compare_float_avx2(values, count, literal, op, matches); + return; + } +#endif + raw_compare_scalar(values, count, literal, op, matches); +} + +void raw_compare(const uint8_t* values, size_t count, double literal, RawComparisonOp op, + uint8_t* matches) { +#ifdef DORIS_PARQUET_X86_SIMD + if (has_avx2() && count >= 4) { + raw_compare_double_avx2(values, count, literal, op, matches); + return; + } +#endif + raw_compare_scalar(values, count, literal, op, matches); +} + +} // namespace doris::simd diff --git a/be/src/util/simd/parquet_kernels.h b/be/src/util/simd/parquet_kernels.h new file mode 100644 index 00000000000000..2381b85e41cb01 --- /dev/null +++ b/be/src/util/simd/parquet_kernels.h @@ -0,0 +1,48 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +#pragma once + +#include +#include + +namespace doris::simd { + +enum class RawComparisonOp : uint8_t { EQ, NE, LT, LE, GT, GE }; + +bool try_byte_stream_split_decode(const uint8_t* src, size_t width, size_t offset, + size_t num_values, size_t stride, uint8_t* dest); + +void delta_decode(int32_t* values, size_t count, int32_t min_delta, int32_t* last_value); +void delta_decode(int64_t* values, size_t count, int64_t min_delta, int64_t* last_value); + +void dictionary_gather(const uint8_t* dictionary, const uint32_t* indices, size_t count, + size_t value_width, uint8_t* dest); + +void expand_nullable_values(uint8_t* values, size_t compact_count, const uint8_t* nulls, + size_t output_count, size_t value_width); + +void raw_compare(const uint8_t* values, size_t count, int32_t literal, RawComparisonOp op, + uint8_t* matches); +void raw_compare(const uint8_t* values, size_t count, int64_t literal, RawComparisonOp op, + uint8_t* matches); +void raw_compare(const uint8_t* values, size_t count, float literal, RawComparisonOp op, + uint8_t* matches); +void raw_compare(const uint8_t* values, size_t count, double literal, RawComparisonOp op, + uint8_t* matches); + +} // namespace doris::simd diff --git a/be/test/exec/scan/file_scanner_v2_test.cpp b/be/test/exec/scan/file_scanner_v2_test.cpp index 3f3d13c98e9bac..2950fa4911d9e4 100644 --- a/be/test/exec/scan/file_scanner_v2_test.cpp +++ b/be/test/exec/scan/file_scanner_v2_test.cpp @@ -83,6 +83,7 @@ TFileRangeDesc paimon_cpp_jni_range() { auto range = range_with_format("paimon", TFileFormatType::FORMAT_JNI); TPaimonFileDesc paimon_params; paimon_params.__set_reader_type(TPaimonReaderType::PAIMON_CPP); + paimon_params.__set_file_format("parquet"); range.table_format_params.__set_paimon_params(std::move(paimon_params)); return range; } @@ -299,7 +300,9 @@ TEST(FileScannerV2Test, SupportedFormatMatrix) { {"remote_doris", TFileFormatType::FORMAT_ARROW, std::nullopt, true}, {"hive", TFileFormatType::FORMAT_ARROW, std::nullopt, false}, {"", TFileFormatType::FORMAT_ARROW, std::nullopt, false}, - {"", TFileFormatType::FORMAT_WAL, std::nullopt, false}, + {"", TFileFormatType::FORMAT_WAL, std::nullopt, true}, + {"", TFileFormatType::FORMAT_ES_HTTP, std::nullopt, false}, + {"", TFileFormatType::FORMAT_LANCE, std::nullopt, false}, }; for (const auto& test_case : cases) { @@ -382,9 +385,13 @@ TEST(FileScannerV2Test, FileScanLocalStateSelectsV2ForSupportedQueriesOnly) { EXPECT_TRUE(FileScanLocalState::TEST_should_use_file_scanner_v2(query_options, false, params)); EXPECT_FALSE(FileScanLocalState::TEST_should_use_file_scanner_v2(query_options, true, params)); - const std::vector unsupported_formats { - TFileFormatType::FORMAT_WAL, - }; + params.__set_format_type(TFileFormatType::FORMAT_WAL); + EXPECT_TRUE(FileScanLocalState::TEST_should_use_file_scanner_v2(query_options, false, params)); + params.__set_format_type(TFileFormatType::FORMAT_JNI); + EXPECT_TRUE(FileScanLocalState::TEST_should_use_file_scanner_v2(query_options, false, params)); + + const std::vector unsupported_formats {TFileFormatType::FORMAT_ES_HTTP, + TFileFormatType::FORMAT_LANCE}; for (const auto format : unsupported_formats) { params.__set_format_type(format); EXPECT_FALSE( @@ -404,24 +411,23 @@ TEST(FileScannerV2Test, FileScanLocalStateSelectsV2ForSupportedQueriesOnly) { EXPECT_FALSE(FileScanLocalState::TEST_should_use_file_scanner_v2(query_options, false, params)); } -TEST(FileScannerV2Test, JniCompatibilityShapesForceLegacyScanner) { +TEST(FileScannerV2Test, JniCompatibilityShapesUseV2Scanner) { TQueryOptions query_options; query_options.__set_enable_file_scanner_v2(true); query_options.__set_enable_paimon_cpp_reader(true); TFileScanRangeParams params; params.__set_format_type(TFileFormatType::FORMAT_JNI); - // Rolling upgrades may carry the only Paimon marker and reader type on each split. Since the - // scan-level selector cannot inspect that split yet, JNI scans conservatively stay on V1. - EXPECT_FALSE(FileScanLocalState::TEST_should_use_file_scanner_v2(query_options, false, params)); - EXPECT_FALSE(FileScannerV2::is_supported(params, paimon_cpp_jni_range())); + EXPECT_TRUE(FileScanLocalState::TEST_should_use_file_scanner_v2(query_options, false, params)); + const auto cpp_range = paimon_cpp_jni_range(); + EXPECT_FALSE(FileScannerV2::is_supported(params, cpp_range)); + const auto cpp_status = FileScannerV2::TEST_validate_scan_range(params, cpp_range); + EXPECT_TRUE(cpp_status.is()); - // Older FEs can omit reader_type. The legacy scanner interprets this as Paimon JNI when the C++ - // reader is disabled, so the scan-level choice must still stay on V1. + // Older FE plans without reader_type used Java whenever the C++ option was disabled. query_options.__set_enable_paimon_cpp_reader(false); - EXPECT_FALSE(FileScanLocalState::TEST_should_use_file_scanner_v2(query_options, false, params)); - EXPECT_FALSE( - FileScannerV2::is_supported(params, legacy_paimon_jni_range_without_reader_type())); + EXPECT_TRUE(FileScanLocalState::TEST_should_use_file_scanner_v2(query_options, false, params)); + EXPECT_TRUE(FileScannerV2::is_supported(params, legacy_paimon_jni_range_without_reader_type())); } TEST(FileScannerV2Test, FailedTableReaderCloseCanBeRetriedThroughScanner) { @@ -477,6 +483,7 @@ TEST(FileScannerV2Test, FileFormatConversionMatrix) { {TFileFormatType::FORMAT_JSON, format::FileFormat::JSON}, {TFileFormatType::FORMAT_NATIVE, format::FileFormat::NATIVE}, {TFileFormatType::FORMAT_ARROW, format::FileFormat::ARROW}, + {TFileFormatType::FORMAT_WAL, format::FileFormat::WAL}, {TFileFormatType::FORMAT_ORC, format::FileFormat::ORC}, }; @@ -659,6 +666,18 @@ TEST(FileScannerV2Test, EndOfFileIsSkippedAsEmptySplit) { EXPECT_FALSE(FileScannerV2::TEST_should_skip_empty(Status::OK(), false)); } +TEST(FileScannerV2Test, OrcScannerResidualFilterRetainsNextBatchContext) { + auto status = FileScannerV2::TEST_contextualize_output_filter_status( + Status::InvalidArgument("synthetic row filter failure"), TFileFormatType::FORMAT_ORC); + EXPECT_NE(status.to_string().find("nextBatch failed"), std::string::npos) << status; + EXPECT_NE(status.to_string().find("synthetic row filter failure"), std::string::npos) << status; + + status = FileScannerV2::TEST_contextualize_output_filter_status( + Status::InvalidArgument("synthetic row filter failure"), + TFileFormatType::FORMAT_PARQUET); + EXPECT_EQ(status.to_string().find("nextBatch failed"), std::string::npos) << status; +} + // Scenario: partition slots are identified from the explicit FE category when present, otherwise // from the legacy is_file_slot flag. Scanner-generated rowid columns must never be treated as // partition columns even if FE marks them as non-file slots. @@ -785,27 +804,4 @@ TEST(FileScannerTest, PartitionPruningStopsAtUnsafePredicate) { EXPECT_EQ(partition_conjuncts[0], conjuncts[0]); } -TEST(FileScannerV2Test, ScannerOwnsUnsafeConjunctAndOrderedSuffixInProfile) { - const auto bool_type = std::make_shared(); - auto unsafe_predicate = std::make_shared(); - unsafe_predicate->add_child(slot_ref(1, 0, bool_type, "part")); - VExprContextSPtrs conjuncts { - runtime_filter_context(slot_ref(1, 0, bool_type, "part"), 1), - runtime_filter_context(std::move(unsafe_predicate), 2), - runtime_filter_context(slot_ref(1, 0, bool_type, "part"), 3), - }; - - RuntimeState state {TQueryOptions(), TQueryGlobals()}; - RuntimeProfile profile("file_scanner_v2"); - FileScannerV2 scanner(&state, &profile, nullptr); - scanner.TEST_set_scanner_conjuncts(std::move(conjuncts)); - - EXPECT_EQ(scanner.TEST_table_reader_owned_conjunct_count(), 1); - EXPECT_EQ(scanner.TEST_scanner_residual_conjunct_count(), 2); - const auto* residual_predicates = profile.get_info_string("ScannerResidualPredicates"); - ASSERT_NE(residual_predicates, nullptr); - EXPECT_FALSE(residual_predicates->empty()); - EXPECT_NE(residual_predicates->find("SlotRef"), std::string::npos) << *residual_predicates; -} - } // namespace doris diff --git a/be/test/exec/scan/scanner_late_arrival_rf_test.cpp b/be/test/exec/scan/scanner_late_arrival_rf_test.cpp index c926f3dc58d595..c65f3bd6f5ce07 100644 --- a/be/test/exec/scan/scanner_late_arrival_rf_test.cpp +++ b/be/test/exec/scan/scanner_late_arrival_rf_test.cpp @@ -68,23 +68,6 @@ class TestScanner final : public Scanner { std::list _blocks; }; -class HighCostPredicate final : public VExpr { -public: - HighCostPredicate() : VExpr(std::make_shared(), false) {} - - Status execute_column_impl(VExprContext*, const Block*, const Selector*, size_t count, - ColumnPtr& result_column) const override { - result_column = ColumnUInt8::create(count, 1); - return Status::OK(); - } - - const std::string& expr_name() const override { return _expr_name; } - double execute_cost() const override { return 100.0; } - -private: - const std::string _expr_name = "high_cost_stateful_predicate"; -}; - class ScannerLateArrivalRfTest : public RuntimeFilterTest { public: void SetUp() override { @@ -103,7 +86,6 @@ class ScannerLateArrivalRfTest : public RuntimeFilterTest { // the counter advances after RFs arrive and that the second call short-circuits // via the fast path at the top of the function. TEST_F(ScannerLateArrivalRfTest, applied_rf_num_advances_after_late_arrival) { - _runtime_states[0]->_query_options.__set_enable_adjust_conjunct_order_by_cost(true); std::vector rf_descs = { TRuntimeFilterDescBuilder().add_planId_to_target_expr(0).build(), TRuntimeFilterDescBuilder().add_planId_to_target_expr(0).build()}; @@ -125,55 +107,26 @@ TEST_F(ScannerLateArrivalRfTest, applied_rf_num_advances_after_late_arrival) { auto local_state = std::make_shared(_runtime_states[0].get(), op.get()); - auto initial_conjunct = VExprContext::create_shared(std::make_shared()); - ASSERT_TRUE(initial_conjunct->prepare(_runtime_states[0].get(), row_desc).ok()); - ASSERT_TRUE(initial_conjunct->open(_runtime_states[0].get()).ok()); - local_state->_conjuncts.push_back(initial_conjunct); - std::vector> rf_dependencies; ASSERT_TRUE(local_state->_helper.init(_runtime_states[0].get(), true, 0, 0, rf_dependencies, "") .ok()); - std::shared_ptr producer; - ASSERT_TRUE(RuntimeFilterProducer::create(_query_ctx.get(), rf_descs.data(), &producer).ok()); - producer->set_wrapper_state_and_ready_to_publish(RuntimeFilterWrapper::State::READY); - local_state->_helper._consumers[0]->signal(producer.get()); - ASSERT_TRUE(local_state->_helper - .acquire_runtime_filter(_runtime_states[0].get(), local_state->_conjuncts, - row_desc) - .ok()); - ASSERT_EQ(local_state->_conjuncts.size(), 2); - auto scanner = std::make_unique(_runtime_states[0].get(), local_state.get(), -1 /*limit*/, &_profile); - ASSERT_TRUE(scanner->init(_runtime_states[0].get(), local_state->_conjuncts).ok()); - auto second_scanner = std::make_unique(_runtime_states[0].get(), local_state.get(), - -1 /*limit*/, &_profile); - ASSERT_TRUE(second_scanner->init(_runtime_states[0].get(), local_state->_conjuncts).ok()); + ASSERT_TRUE(scanner->init(_runtime_states[0].get(), {}).ok()); ASSERT_EQ(scanner->_total_rf_num, 2); ASSERT_EQ(scanner->_applied_rf_num, 0); + std::shared_ptr producer; + ASSERT_TRUE(RuntimeFilterProducer::create(_query_ctx.get(), rf_descs.data(), &producer).ok()); + producer->set_wrapper_state_and_ready_to_publish(RuntimeFilterWrapper::State::READY); + local_state->_helper._consumers[0]->signal(producer.get()); local_state->_helper._consumers[1]->signal(producer.get()); // First call after both RFs arrived: counter must advance to total. Before // the fix this stayed at 0 because the assignment was missing. ASSERT_TRUE(scanner->try_append_late_arrival_runtime_filter().ok()); ASSERT_EQ(scanner->_applied_rf_num, 2); - ASSERT_EQ(scanner->_late_arrival_rf_conjuncts.size(), 1); - for (const auto& conjunct : scanner->_late_arrival_rf_conjuncts) { - EXPECT_NE(dynamic_cast(conjunct->root().get()), nullptr); - } - ASSERT_EQ(scanner->_conjuncts.size(), 3); - EXPECT_EQ(scanner->_conjuncts.back()->expr_name(), "high_cost_stateful_predicate"); - - // The first scanner consumes the shared helper's expression, so another scanner can only get - // the exact delta from the local state's append-only RF batch history. - ASSERT_TRUE(second_scanner->try_append_late_arrival_runtime_filter().ok()); - ASSERT_EQ(second_scanner->_applied_rf_num, 2); - ASSERT_EQ(second_scanner->_late_arrival_rf_conjuncts.size(), 1); - EXPECT_NE(dynamic_cast( - second_scanner->_late_arrival_rf_conjuncts[0]->root().get()), - nullptr); // Second call: must hit the fast-path early return without re-cloning. // We clear `_conjuncts` and verify the function does NOT repopulate them; diff --git a/be/test/exprs/bloom_filter_func_test.cpp b/be/test/exprs/bloom_filter_func_test.cpp index bc0ce276a37e7e..741181c6256cf0 100644 --- a/be/test/exprs/bloom_filter_func_test.cpp +++ b/be/test/exprs/bloom_filter_func_test.cpp @@ -20,6 +20,7 @@ #include #include +#include #include #include @@ -200,6 +201,134 @@ TEST_F(BloomFilterFuncTest, InsertFixedLen) { ASSERT_EQ(offsets[1], 2); } +TEST_F(BloomFilterFuncTest, RawFixedCapabilitiesCoverEveryFixedRuntimeFilterType) { +#define EXPECT_RAW_FIXED(TYPE) \ + do { \ + BloomFilterFunc filter(false); \ + EXPECT_TRUE(filter.supports_raw_fixed_values()) << type_to_string(TYPE); \ + EXPECT_EQ(filter.raw_fixed_value_size(), \ + sizeof(typename PrimitiveTypeTraits::CppType)) \ + << type_to_string(TYPE); \ + } while (false) + EXPECT_RAW_FIXED(TYPE_BOOLEAN); + EXPECT_RAW_FIXED(TYPE_TINYINT); + EXPECT_RAW_FIXED(TYPE_SMALLINT); + EXPECT_RAW_FIXED(TYPE_INT); + EXPECT_RAW_FIXED(TYPE_BIGINT); + EXPECT_RAW_FIXED(TYPE_LARGEINT); + EXPECT_RAW_FIXED(TYPE_FLOAT); + EXPECT_RAW_FIXED(TYPE_DOUBLE); + EXPECT_RAW_FIXED(TYPE_DATE); + EXPECT_RAW_FIXED(TYPE_DATETIME); + EXPECT_RAW_FIXED(TYPE_DATEV2); + EXPECT_RAW_FIXED(TYPE_DATETIMEV2); + EXPECT_RAW_FIXED(TYPE_TIMESTAMPTZ); + EXPECT_RAW_FIXED(TYPE_DECIMAL32); + EXPECT_RAW_FIXED(TYPE_DECIMAL64); + EXPECT_RAW_FIXED(TYPE_DECIMALV2); + EXPECT_RAW_FIXED(TYPE_DECIMAL128I); + EXPECT_RAW_FIXED(TYPE_DECIMAL256); + EXPECT_RAW_FIXED(TYPE_IPV4); + EXPECT_RAW_FIXED(TYPE_IPV6); +#undef EXPECT_RAW_FIXED + + BloomFilterFunc string_filter(false); + EXPECT_FALSE(string_filter.supports_raw_fixed_values()); + EXPECT_EQ(string_filter.raw_fixed_value_size(), 0); +} + +TEST_F(BloomFilterFuncTest, RawFixedProbeUsesTheSameHashAsColumnProbe) { + BloomFilterFunc filter(false); + RuntimeFilterParams params { + 1, RuntimeFilterType::BLOOM_FILTER, TYPE_INT, false, 0, 0, 0, 256, 0, 0}; + filter.init_params(¶ms); + ASSERT_TRUE(filter.init_with_fixed_length(1024).ok()); + auto build_column = ColumnHelper::create_column({2, 4}); + filter.insert_fixed_len(build_column, 0); + + const std::array values {1, 2, 3, 4}; + std::array raw_matches {1, 1, 1, 1}; + ASSERT_TRUE(filter.find_batch_raw_fixed(reinterpret_cast(values.data()), + values.size(), sizeof(int32_t), raw_matches.data()) + .ok()); + + auto probe_column = ColumnHelper::create_column({1, 2, 3, 4}); + std::array column_matches {}; + filter.find_fixed_len(probe_column, column_matches.data()); + EXPECT_EQ(column_matches, raw_matches); +} + +TEST_F(BloomFilterFuncTest, RawFixedProbeMatchesBuildHashForEveryFixedRuntimeFilterType) { + const auto expect_match = []() { + BloomFilterFunc filter(false); + RuntimeFilterParams params; + params.filter_type = RuntimeFilterType::BLOOM_FILTER; + params.column_return_type = TYPE; + params.bloom_filter_size = 1024; + filter.init_params(¶ms); + ASSERT_TRUE(filter.init_with_fixed_length(1024).ok()) << type_to_string(TYPE); + + using ValueType = typename PrimitiveTypeTraits::CppType; + ValueType value {}; + auto set = std::make_shared>(false); + set->insert(&value); + filter.insert_set(set); + + uint8_t match = 1; + ASSERT_TRUE(filter.find_batch_raw_fixed(reinterpret_cast(&value), 1, + sizeof(ValueType), &match) + .ok()) + << type_to_string(TYPE); + EXPECT_EQ(match, 1) << type_to_string(TYPE); + EXPECT_TRUE(filter.test_field(Field::create_field(value))) << type_to_string(TYPE); + }; + +#define EXPECT_RAW_FIXED_MATCH(TYPE) expect_match.template operator()() + EXPECT_RAW_FIXED_MATCH(TYPE_BOOLEAN); + EXPECT_RAW_FIXED_MATCH(TYPE_TINYINT); + EXPECT_RAW_FIXED_MATCH(TYPE_SMALLINT); + EXPECT_RAW_FIXED_MATCH(TYPE_INT); + EXPECT_RAW_FIXED_MATCH(TYPE_BIGINT); + EXPECT_RAW_FIXED_MATCH(TYPE_LARGEINT); + EXPECT_RAW_FIXED_MATCH(TYPE_FLOAT); + EXPECT_RAW_FIXED_MATCH(TYPE_DOUBLE); + EXPECT_RAW_FIXED_MATCH(TYPE_DATE); + EXPECT_RAW_FIXED_MATCH(TYPE_DATETIME); + EXPECT_RAW_FIXED_MATCH(TYPE_DATEV2); + EXPECT_RAW_FIXED_MATCH(TYPE_DATETIMEV2); + EXPECT_RAW_FIXED_MATCH(TYPE_TIMESTAMPTZ); + EXPECT_RAW_FIXED_MATCH(TYPE_DECIMAL32); + EXPECT_RAW_FIXED_MATCH(TYPE_DECIMAL64); + EXPECT_RAW_FIXED_MATCH(TYPE_DECIMALV2); + EXPECT_RAW_FIXED_MATCH(TYPE_DECIMAL128I); + EXPECT_RAW_FIXED_MATCH(TYPE_DECIMAL256); + EXPECT_RAW_FIXED_MATCH(TYPE_IPV4); + EXPECT_RAW_FIXED_MATCH(TYPE_IPV6); +#undef EXPECT_RAW_FIXED_MATCH +} + +TEST_F(BloomFilterFuncTest, DictionaryFieldProbeSupportsEveryStringRuntimeFilterType) { + const auto expect_match = []() { + BloomFilterFunc filter(false); + RuntimeFilterParams params; + params.filter_type = RuntimeFilterType::BLOOM_FILTER; + params.column_return_type = TYPE; + params.bloom_filter_size = 1024; + filter.init_params(¶ms); + ASSERT_TRUE(filter.init_with_fixed_length(1024).ok()) << type_to_string(TYPE); + + auto column = ColumnString::create(); + column->insert_data("value", 5); + filter.insert_fixed_len(std::move(column), 0); + EXPECT_TRUE(filter.test_field(Field::create_field(std::string("value")))) + << type_to_string(TYPE); + }; + + expect_match.template operator()(); + expect_match.template operator()(); + expect_match.template operator()(); +} + TEST_F(BloomFilterFuncTest, Merge) { BloomFilterFunc bloom_filter_func(false); const size_t runtime_length = 1024; diff --git a/be/test/exprs/expr_zonemap_filter_test.cpp b/be/test/exprs/expr_zonemap_filter_test.cpp index 338d1de79fbfc0..1d80f40531e663 100644 --- a/be/test/exprs/expr_zonemap_filter_test.cpp +++ b/be/test/exprs/expr_zonemap_filter_test.cpp @@ -19,6 +19,7 @@ #include +#include #include #include #include @@ -26,6 +27,7 @@ #include #include "common/object_pool.h" +#include "core/column/column_vector.h" #include "core/data_type/data_type_date_or_datetime_v2.h" #include "core/data_type/data_type_decimal.h" #include "core/data_type/data_type_nullable.h" @@ -34,10 +36,13 @@ #include "core/field.h" #include "core/string_ref.h" #include "core/value/vdatetime_value.h" +#include "exprs/bloom_filter_func.h" #include "exprs/create_predicate_function.h" #include "exprs/function/functions_comparison.h" #include "exprs/function/simple_function_factory.h" #include "exprs/hybrid_set.h" +#include "exprs/runtime_filter_expr.h" +#include "exprs/vbloom_predicate.h" #include "exprs/vcompound_pred.h" #include "exprs/vexpr.h" #include "exprs/vexpr_context.h" @@ -368,7 +373,7 @@ TEST(ExprZonemapFilterTest, ComparisonZonemapHandlesNullAndUnsupportedInputs) { EXPECT_EQ(1, pass_all_ctx.stats.unusable_zonemap_eval_count); } -TEST(ExprZonemapFilterTest, ComparisonDictionaryAndBloomUseEqualityLiterals) { +TEST(ExprZonemapFilterTest, ComparisonDictionarySupportsTypedRangesWhileBloomUsesEquality) { auto type = int_type(); auto slot = make_slot(0, type); FunctionComparison equals; @@ -389,8 +394,37 @@ TEST(ExprZonemapFilterTest, ComparisonDictionaryAndBloomUseEqualityLiterals) { equals.evaluate_bloom_filter(bloom_ctx, {slot, make_int_literal(3)})); FunctionComparison not_equals; - EXPECT_FALSE(not_equals.can_evaluate_dictionary_filter({slot, make_int_literal(3)})); + FunctionComparison less; + FunctionComparison less_equal; + FunctionComparison greater; + FunctionComparison greater_equal; + EXPECT_TRUE(not_equals.can_evaluate_dictionary_filter({slot, make_int_literal(3)})); + EXPECT_TRUE(less.can_evaluate_dictionary_filter({slot, make_int_literal(2)})); + EXPECT_TRUE(less_equal.can_evaluate_dictionary_filter({slot, make_int_literal(1)})); + EXPECT_TRUE(greater.can_evaluate_dictionary_filter({slot, make_int_literal(2)})); + EXPECT_TRUE(greater_equal.can_evaluate_dictionary_filter({slot, make_int_literal(3)})); + EXPECT_EQ(ZoneMapFilterResult::kMayMatch, + less.evaluate_dictionary_filter(dictionary_ctx, {slot, make_int_literal(2)})); + EXPECT_EQ(ZoneMapFilterResult::kNoMatch, + less.evaluate_dictionary_filter(dictionary_ctx, {slot, make_int_literal(1)})); + EXPECT_EQ(ZoneMapFilterResult::kMayMatch, + less.evaluate_dictionary_filter(dictionary_ctx, {make_int_literal(2), slot})); + EXPECT_EQ(ZoneMapFilterResult::kNoMatch, + less.evaluate_dictionary_filter(dictionary_ctx, {make_int_literal(4), slot})); EXPECT_FALSE(not_equals.can_evaluate_bloom_filter({slot, make_int_literal(3)})); + + auto string_type = std::make_shared(); + auto string_slot = make_slot(0, string_type); + auto string_dictionary = make_dictionary_context({Field::create_field("alpha"), + Field::create_field("charlie")}, + string_type); + EXPECT_TRUE(less.can_evaluate_dictionary_filter({string_slot, make_string_literal("bravo")})); + EXPECT_EQ(ZoneMapFilterResult::kMayMatch, + less.evaluate_dictionary_filter(string_dictionary, + {string_slot, make_string_literal("bravo")})); + EXPECT_EQ(ZoneMapFilterResult::kNoMatch, + greater.evaluate_dictionary_filter(string_dictionary, + {string_slot, make_string_literal("delta")})); } TEST(ExprZonemapFilterTest, DefaultFunctionForwardsDictionaryAndBloomEvaluation) { @@ -879,6 +913,97 @@ TEST(ExprZonemapFilterTest, RuntimeFilterWrapperNullAwareZonemapKeepsZonesWithNu runtime_filter->evaluate_zonemap_filter(only_null_ctx)); } +TEST(ExprZonemapFilterTest, RuntimeFilterExprDelegatesDirectInDictionaryAndRawEvaluation) { + auto type = int_type(); + auto slot = make_slot(0, type); + std::shared_ptr filter(create_set(PrimitiveType::TYPE_INT, false)); + int32_t two = 2; + int32_t four = 4; + filter->insert(&two); + filter->insert(&four); + + auto direct_in_expr = + std::make_shared(make_in_predicate_node(false, 1), filter, true); + direct_in_expr->add_child(slot); + ASSERT_TRUE(direct_in_expr->_materialize_for_zonemap_filter().ok()); + + auto runtime_filter = RuntimeFilterExpr::create_shared(make_in_predicate_node(false, 1), + direct_in_expr, 0.0, false, 7); + EXPECT_TRUE(runtime_filter->can_evaluate_dictionary_filter()); + EXPECT_EQ(ZoneMapFilterResult::kNoMatch, + runtime_filter->evaluate_dictionary_filter( + make_dictionary_context({int_field(1), int_field(3)}, type))); + EXPECT_EQ(ZoneMapFilterResult::kMayMatch, + runtime_filter->evaluate_dictionary_filter( + make_dictionary_context({int_field(3), int_field(4)}, type))); + + EXPECT_TRUE(runtime_filter->can_execute_on_raw_fixed_values(type, 0)); + const std::array values {1, 2, 3, 4}; + std::array matches {1, 1, 1, 1}; + ASSERT_TRUE(runtime_filter + ->execute_on_raw_fixed_values( + reinterpret_cast(values.data()), values.size(), + sizeof(int32_t), type, 0, matches.data()) + .ok()); + EXPECT_EQ((std::array {0, 1, 0, 1}), matches); + + auto null_aware_runtime_filter = RuntimeFilterExpr::create_shared( + make_in_predicate_node(false, 1), direct_in_expr, 0.0, true, 8); + EXPECT_FALSE(null_aware_runtime_filter->can_evaluate_dictionary_filter()); + EXPECT_FALSE(null_aware_runtime_filter->can_execute_on_raw_fixed_values(type, 0)); +} + +TEST(ExprZonemapFilterTest, RuntimeFilterExprDelegatesBloomDictionaryAndRawEvaluation) { + auto type = int_type(); + std::shared_ptr filter(create_bloom_filter(TYPE_INT, false)); + RuntimeFilterParams params; + params.filter_type = RuntimeFilterType::BLOOM_FILTER; + params.column_return_type = TYPE_INT; + params.bloom_filter_size = 1024; + filter->init_params(¶ms); + ASSERT_TRUE(filter->init_with_fixed_length(1024).ok()); + auto build_values = ColumnInt32::create(); + build_values->insert_value(2); + build_values->insert_value(4); + filter->insert_fixed_len(std::move(build_values), 0); + + auto node = make_in_predicate_node(false, 1); + node.__set_node_type(TExprNodeType::BLOOM_PRED); + node.__set_opcode(TExprOpcode::RT_FILTER); + auto bloom = VBloomPredicate::create_shared(node); + bloom->set_filter(filter); + bloom->add_child(make_slot(0, type)); + auto runtime_filter = RuntimeFilterExpr::create_shared(node, bloom, 0.0, false, 9); + + EXPECT_TRUE(runtime_filter->can_evaluate_dictionary_filter()); + EXPECT_EQ(ZoneMapFilterResult::kMayMatch, + runtime_filter->evaluate_dictionary_filter( + make_dictionary_context({int_field(2)}, type))); + + int32_t missing = 1; + while (missing < 10000 && filter->test_field(int_field(missing))) { + ++missing; + } + ASSERT_LT(missing, 10000); + EXPECT_EQ(ZoneMapFilterResult::kNoMatch, + runtime_filter->evaluate_dictionary_filter( + make_dictionary_context({int_field(missing)}, type))); + + EXPECT_TRUE(runtime_filter->can_execute_on_raw_fixed_values(type, 0)); + const std::array values {missing, 2, 4}; + std::array matches {1, 1, 1}; + ASSERT_TRUE(runtime_filter + ->execute_on_raw_fixed_values( + reinterpret_cast(values.data()), values.size(), + sizeof(int32_t), type, 0, matches.data()) + .ok()); + EXPECT_EQ((std::array {0, 1, 1}), matches); + + auto null_aware_runtime_filter = RuntimeFilterExpr::create_shared(node, bloom, 0.0, true, 10); + EXPECT_FALSE(null_aware_runtime_filter->can_evaluate_dictionary_filter()); + EXPECT_FALSE(null_aware_runtime_filter->can_execute_on_raw_fixed_values(type, 0)); +} + TEST(ExprZonemapFilterTest, CompoundPredicateEvaluatesChildrenForZonemap) { ZoneMapEvalContext ctx; diff --git a/be/test/format/table/iceberg/iceberg_position_delete_sys_table_reader_test.cpp b/be/test/format/table/iceberg/iceberg_position_delete_sys_table_reader_test.cpp index f3c5ef1c386c8f..ee041bab621785 100644 --- a/be/test/format/table/iceberg/iceberg_position_delete_sys_table_reader_test.cpp +++ b/be/test/format/table/iceberg/iceberg_position_delete_sys_table_reader_test.cpp @@ -29,7 +29,6 @@ #include "core/column/column_nullable.h" #include "core/column/column_string.h" #include "core/column/column_struct.h" -#include "core/column/column_vector.h" #include "core/data_type/data_type_nullable.h" #include "core/data_type/data_type_number.h" #include "core/data_type/data_type_string.h" @@ -75,31 +74,6 @@ class FillColumnsTrackingReader final : public GenericReader { } }; -class RejectAllRowsPredicate final : public VExpr { -public: - RejectAllRowsPredicate() : VExpr(std::make_shared(), false) {} - - Status execute_column_impl(VExprContext*, const Block*, const Selector*, size_t count, - ColumnPtr& result_column) const override { - auto result = ColumnUInt8::create(); - result->get_data().resize_fill(count, 0); - result_column = std::move(result); - return Status::OK(); - } - - const std::string& expr_name() const override { return _name; } - bool is_deterministic() const override { return false; } - - Status clone_node(VExprSPtr* cloned_expr) const override { - DORIS_CHECK(cloned_expr != nullptr); - *cloned_expr = std::make_shared(); - return Status::OK(); - } - -private: - const std::string _name = "RejectAllRowsPredicate"; -}; - SlotDescriptor* make_slot(ObjectPool* pool, int id, std::string name, DataTypePtr type) { TSlotDescriptor slot_desc; slot_desc.__set_id(id); @@ -671,50 +645,4 @@ TEST(IcebergPositionDeleteSysTableV2ReaderTest, StopsBeforeExpandingDeletionVect EXPECT_TRUE(eof); } -TEST(IcebergPositionDeleteSysTableV2ReaderTest, - AllFilteredDeletionVectorYieldsBeforeObservingCancellation) { - ObjectPool pool; - RuntimeState state {TQueryOptions(), TQueryGlobals()}; - RuntimeProfile profile("test_profile"); - const auto nullable_int64 = make_nullable(std::make_shared()); - std::vector file_slot_descs { - make_slot(&pool, 0, "pos", nullable_int64), - }; - - auto conjunct = VExprContext::create_shared(std::make_shared()); - RowDescriptor row_desc; - ASSERT_TRUE(conjunct->prepare(&state, row_desc).ok()); - ASSERT_TRUE(conjunct->open(&state).ok()); - - format::iceberg::IcebergPositionDeleteSysTableV2Reader reader; - reader._runtime_state = &state; - reader._scanner_profile = &profile; - reader._io_ctx = std::make_shared(); - reader._file_slot_descs = &file_slot_descs; - reader._projected_columns.resize(file_slot_descs.size()); - reader._remaining_conjuncts = {conjunct}; - reader._has_split = true; - reader._delete_file_kind = - format::iceberg::IcebergPositionDeleteSysTableV2Reader::DeleteFileKind::DELETION_VECTOR; - reader._batch_size = 1; - reader._dv_positions.add(uint64_t {7}); - reader._dv_positions.add(uint64_t {9}); - reader._dv_positions.add(uint64_t {11}); - reader._next_dv_position.emplace(reader._dv_positions.begin()); - - Block block = make_output_block(file_slot_descs); - bool eof = false; - ASSERT_TRUE(reader.get_block(&block, &eof).ok()); - EXPECT_FALSE(eof); - EXPECT_EQ(block.rows(), 0); - ASSERT_TRUE(reader._next_dv_position.has_value()); - EXPECT_EQ(**reader._next_dv_position, 9); - - reader._io_ctx->should_stop = true; - ASSERT_TRUE(reader.get_block(&block, &eof).ok()); - EXPECT_TRUE(eof); - ASSERT_TRUE(reader._next_dv_position.has_value()); - EXPECT_EQ(**reader._next_dv_position, 9); -} - } // namespace doris diff --git a/be/test/format_v2/column_mapper_test.cpp b/be/test/format_v2/column_mapper_test.cpp index d3428b6134a479..29d4efbe17f44e 100644 --- a/be/test/format_v2/column_mapper_test.cpp +++ b/be/test/format_v2/column_mapper_test.cpp @@ -2337,39 +2337,6 @@ TEST(ColumnMapperLocalizeFiltersTest, VisibleLocalFilterAddsPredicateColumnAndCo EXPECT_TRUE(localized_slot->data_type()->equals(*int_type)); } -TEST(ColumnMapperLocalizeFiltersTest, ReportsLocalizationForEachSplitMapping) { - const auto int_type = i32(); - auto table_column = name_col("id", int_type); - const std::vector table_schema = {table_column}; - TableFilter filter { - .conjunct = VExprContext::create_shared(int_gt(table_slot(0, 0, int_type, "id"), 1)), - .global_indices = {GlobalIndex(0)}}; - - TableColumnMapper local_mapper({.mode = TableColumnMappingMode::BY_NAME}); - ASSERT_TRUE(local_mapper.create_mapping(table_schema, {}, {name_col("id", int_type, 7)}).ok()); - FileScanRequest local_request; - FilterLocalizationResult local_result; - ASSERT_TRUE(local_mapper - .create_scan_request({filter}, table_schema, &local_request, nullptr, - &local_result) - .ok()); - ASSERT_EQ(local_result.localized_filters.size(), 1); - EXPECT_TRUE(local_result.localized_filters[0]); - ASSERT_EQ(local_request.conjuncts.size(), 1); - - TableColumnMapper missing_mapper({.mode = TableColumnMappingMode::BY_NAME}); - ASSERT_TRUE(missing_mapper.create_mapping(table_schema, {}, {}).ok()); - FileScanRequest missing_request; - FilterLocalizationResult missing_result; - ASSERT_TRUE(missing_mapper - .create_scan_request({filter}, table_schema, &missing_request, nullptr, - &missing_result) - .ok()); - ASSERT_EQ(missing_result.localized_filters.size(), 1); - EXPECT_FALSE(missing_result.localized_filters[0]); - EXPECT_TRUE(missing_request.conjuncts.empty()); -} - TEST(ColumnMapperLocalizeFiltersTest, VarbinaryFilterStaysAboveFileReader) { const auto binary_type = varbinary(); const auto table_column = name_col("partition_key", binary_type); @@ -2395,35 +2362,6 @@ TEST(ColumnMapperLocalizeFiltersTest, VarbinaryFilterStaysAboveFileReader) { EXPECT_TRUE(request.conjuncts.empty()); } -TEST(ColumnMapperLocalizeFiltersTest, VarcharWidthTruncationFilterStaysAboveFileReader) { - const auto table_type = std::make_shared(3, TYPE_VARCHAR); - const auto file_type = std::make_shared(10, TYPE_VARCHAR); - const auto table_column = name_col("value", table_type); - const auto file_column = name_col("value", file_type, 7); - - TableColumnMapper mapper({.mode = TableColumnMappingMode::BY_NAME}); - ASSERT_TRUE(mapper.create_mapping({table_column}, {}, {file_column}).ok()); - - TableFilter filter {.conjunct = VExprContext::create_shared(binary_predicate( - TExprOpcode::EQ, table_slot(0, 0, table_type, "value"), - literal(table_type, Field::create_field("abc")))), - .global_indices = {GlobalIndex(0)}}; - TQueryOptions query_options; - query_options.__set_truncate_char_or_varchar_columns(true); - RuntimeState state {query_options, TQueryGlobals()}; - FileScanRequest request; - FilterLocalizationResult localization_result; - - ASSERT_TRUE(mapper.create_scan_request({filter}, {table_column}, &request, &state, - &localization_result) - .ok()); - ASSERT_EQ(localization_result.localized_filters.size(), 1); - EXPECT_FALSE(localization_result.localized_filters[0]); - EXPECT_TRUE(request.conjuncts.empty()); - ASSERT_EQ(request.non_predicate_columns.size(), 1); - EXPECT_EQ(request.non_predicate_columns[0].column_id(), LocalColumnId(7)); -} - TEST(ColumnMapperLocalizeFiltersTest, NestedVarbinaryFilterStaysAboveFileReader) { const auto table_column = struct_name_col( "payload", {name_col("id", i32()), name_col("binary_value", varbinary())}); @@ -2574,7 +2512,7 @@ TEST(ColumnMapperScanRequestTest, HiddenTopLevelFilterMappingUsesNameFallback) { EXPECT_EQ(mapper.filter_entries().at(GlobalIndex(1)).local_index(), LocalIndex(1)); } -TEST(ColumnMapperScanRequestTest, OrdinaryPredicateSlotRetainsOutputPayload) { +TEST(ColumnMapperScanRequestTest, OrdinaryPredicateSlotRetainsPayloadForScannerBoundary) { const auto int_type = i32(); auto quantity = name_col("ss_quantity", int_type); auto tax = name_col("ss_ext_tax", int_type); @@ -2599,8 +2537,8 @@ TEST(ColumnMapperScanRequestTest, OrdinaryPredicateSlotRetainsOutputPayload) { EXPECT_EQ(request.predicate_columns[0].column_id(), LocalColumnId(0)); ASSERT_EQ(request.non_predicate_columns.size(), 1); EXPECT_EQ(request.non_predicate_columns[0].column_id(), LocalColumnId(1)); - // A visible predicate slot is still part of the table output and cannot be replaced with a - // default-valued placeholder after file-local filtering. + // The scanner evaluates its table-level conjuncts after TableReader returns, so a visible + // predicate slot cannot be replaced with a default-valued placeholder at the file boundary. EXPECT_TRUE(request.predicate_only_columns.empty()); } diff --git a/be/test/format_v2/jni/paimon_jni_reader_test.cpp b/be/test/format_v2/jni/paimon_jni_reader_test.cpp index 570977a5f7eb4e..9c46aba05c76e1 100644 --- a/be/test/format_v2/jni/paimon_jni_reader_test.cpp +++ b/be/test/format_v2/jni/paimon_jni_reader_test.cpp @@ -25,6 +25,7 @@ #include "format_v2/table_reader.h" #include "gen_cpp/PlanNodes_types.h" +#include "runtime/runtime_state.h" namespace doris::format::paimon { namespace { @@ -47,14 +48,15 @@ TFileScanRangeParams make_scan_params() { return scan_params; } -Status init_reader(PaimonJniReader* reader, TFileScanRangeParams* scan_params) { +Status init_reader(PaimonJniReader* reader, TFileScanRangeParams* scan_params, + RuntimeState* runtime_state = nullptr) { return reader->init({ .projected_columns = {}, .conjuncts = {}, .format = FileFormat::JNI, .scan_params = scan_params, .io_ctx = nullptr, - .runtime_state = nullptr, + .runtime_state = runtime_state, .scanner_profile = nullptr, }); } @@ -159,16 +161,20 @@ TEST(PaimonJniReaderTest, ScanLevelOptionsOverrideLegacySplitFallbacks) { EXPECT_EQ(params["hadoop.source"], "scan"); } -TEST(PaimonJniReaderTest, KeepsInitialPhysicalBatchSizeAfterOpen) { +TEST(PaimonJniReaderTest, UsesStableRuntimeBatchSizeBeforeAndAfterOpen) { + TQueryOptions query_options; + query_options.__set_batch_size(8160); + RuntimeState state {query_options, TQueryGlobals()}; + auto scan_params = make_scan_params(); PaimonJniReader reader; + ASSERT_TRUE(init_reader(&reader, &scan_params, &state).ok()); + reader.set_batch_size(32); - EXPECT_EQ(reader.TEST_batch_size(), 32); + EXPECT_EQ(reader.TEST_batch_size(), 8160); - // Paimon copies the constructor size into the RecordReader during Java open. A later predictor - // result cannot resize that physical reader, so keep the initial probe size for the split. reader.TEST_set_split_state(true, false); reader.set_batch_size(1); - EXPECT_EQ(reader.TEST_batch_size(), 32); + EXPECT_EQ(reader.TEST_batch_size(), 8160); } } // namespace diff --git a/be/test/format_v2/parquet/native_decoder_test.cpp b/be/test/format_v2/parquet/native_decoder_test.cpp index 599c930f312856..0d037f61549281 100644 --- a/be/test/format_v2/parquet/native_decoder_test.cpp +++ b/be/test/format_v2/parquet/native_decoder_test.cpp @@ -38,6 +38,7 @@ #include "core/data_type/data_type_string.h" #include "core/data_type/data_type_struct.h" #include "core/data_type/data_type_timestamptz.h" +#include "core/data_type_serde/parquet_decode_source.h" #include "core/data_type_serde/parquet_timestamp.h" #include "exprs/vectorized_fn_call.h" #include "exprs/vliteral.h" @@ -263,6 +264,24 @@ class CaptureFixedConsumer final : public ParquetFixedValueConsumer { std::vector bytes; }; +class CaptureDictionaryConsumer final : public ParquetDictionaryValueConsumer { +public: + Status consume_indices(const uint32_t* values, size_t num_values) override { + ++consume_calls; + indices.insert(indices.end(), values, values + num_values); + return Status::OK(); + } + + Status consume_repeated(uint32_t value, size_t num_values) override { + ++consume_calls; + indices.insert(indices.end(), num_values, value); + return Status::OK(); + } + + size_t consume_calls = 0; + std::vector indices; +}; + TEST(ParquetV2NativeDecoderTest, FragmentedPlainSelectionUsesOneConsumerBatch) { std::array input {}; std::iota(input.begin(), input.end(), 0); @@ -287,6 +306,90 @@ TEST(ParquetV2NativeDecoderTest, FragmentedPlainSelectionUsesOneConsumerBatch) { EXPECT_EQ(consumer.values(), expected); } +TEST(ParquetV2NativeDecoderTest, FragmentedDictionarySelectionUsesOneConsumerBatch) { + constexpr size_t VALUE_COUNT = 32; + std::array dictionary_values {}; + std::iota(dictionary_values.begin(), dictionary_values.end(), 0); + auto dictionary = make_unique_buffer(sizeof(dictionary_values)); + memcpy(dictionary.get(), dictionary_values.data(), sizeof(dictionary_values)); + + std::unique_ptr decoder; + ASSERT_TRUE( + Decoder::get_decoder(tparquet::Type::INT32, tparquet::Encoding::RLE_DICTIONARY, decoder) + .ok()); + decoder->set_type_length(sizeof(int32_t)); + ASSERT_TRUE(decoder->set_dict(dictionary, sizeof(dictionary_values), dictionary_values.size()) + .ok()); + + faststring encoded_ids; + RleEncoder encoder(&encoded_ids, 5); + for (uint32_t id = 0; id < VALUE_COUNT; ++id) { + encoder.Put(id); + } + encoder.Flush(); + std::vector payload(encoded_ids.size() + 1); + payload[0] = 5; + memcpy(payload.data() + 1, encoded_ids.data(), encoded_ids.size()); + Slice data(payload.data(), payload.size()); + ASSERT_TRUE(decoder->set_data(&data).ok()); + + ParquetSelection selection { + .total_values = VALUE_COUNT, .selected_values = VALUE_COUNT / 2, .ranges = {}}; + std::vector expected; + for (uint32_t id = 0; id < VALUE_COUNT; id += 2) { + selection.ranges.push_back({.first = id, .count = 1}); + expected.push_back(id); + } + CaptureDictionaryConsumer consumer; + ASSERT_TRUE(decoder->decode_selected_dictionary_values(selection, consumer).ok()); + + EXPECT_EQ(consumer.consume_calls, 1); + EXPECT_EQ(consumer.indices, expected); + + ASSERT_TRUE(decoder->set_data(&data).ok()); + std::vector selected_indices; + ASSERT_TRUE(decoder->decode_selected_dictionary_indices(selection, &selected_indices).ok()); + EXPECT_EQ(selected_indices, expected); +} + +TEST(ParquetV2NativeDecoderTest, HighlySparseDictionarySelectionAvoidsFullBatchDecode) { + constexpr size_t DICTIONARY_SIZE = 32; + constexpr size_t VALUE_COUNT = 256; + std::array dictionary_values {}; + std::iota(dictionary_values.begin(), dictionary_values.end(), 0); + auto dictionary = make_unique_buffer(sizeof(dictionary_values)); + memcpy(dictionary.get(), dictionary_values.data(), sizeof(dictionary_values)); + + std::unique_ptr decoder; + ASSERT_TRUE( + Decoder::get_decoder(tparquet::Type::INT32, tparquet::Encoding::RLE_DICTIONARY, decoder) + .ok()); + decoder->set_type_length(sizeof(int32_t)); + ASSERT_TRUE(decoder->set_dict(dictionary, sizeof(dictionary_values), DICTIONARY_SIZE).ok()); + + faststring encoded_ids; + RleEncoder encoder(&encoded_ids, 5); + for (uint32_t row = 0; row < VALUE_COUNT; ++row) { + encoder.Put(row % DICTIONARY_SIZE); + } + encoder.Flush(); + std::vector payload(encoded_ids.size() + 1); + payload[0] = 5; + memcpy(payload.data() + 1, encoded_ids.data(), encoded_ids.size()); + Slice data(payload.data(), payload.size()); + ASSERT_TRUE(decoder->set_data(&data).ok()); + + ParquetSelection selection {.total_values = VALUE_COUNT, .selected_values = 8, .ranges = {}}; + for (size_t row = 0; row < selection.selected_values; ++row) { + selection.ranges.push_back({.first = row * 32, .count = 1}); + } + CaptureDictionaryConsumer consumer; + ASSERT_TRUE(decoder->decode_selected_dictionary_values(selection, consumer).ok()); + + EXPECT_EQ(consumer.consume_calls, selection.ranges.size()); + EXPECT_EQ(consumer.indices, std::vector(selection.selected_values, 0)); +} + class ScriptedDictionaryMaterializationSource final : public ParquetDecodeSource { public: ScriptedDictionaryMaterializationSource(std::vector ids, bool prefer_indices) @@ -386,6 +489,7 @@ class MemoryBufferedReader final : public io::BufferedStreamReader { Status read_bytes(const uint8_t** buf, uint64_t offset, size_t bytes_to_read, const io::IOContext*) override { + ++_read_count; if (offset > _data.size() || bytes_to_read > _data.size() - offset) { return Status::IOError("out of bounds"); } @@ -393,6 +497,7 @@ class MemoryBufferedReader final : public io::BufferedStreamReader { return Status::OK(); } Status read_bytes(Slice& slice, uint64_t offset, const io::IOContext*) override { + ++_read_count; if (offset > _data.size() || slice.size > _data.size() - offset) { return Status::IOError("out of bounds"); } @@ -401,9 +506,11 @@ class MemoryBufferedReader final : public io::BufferedStreamReader { } std::string path() override { return "memory.parquet"; } int64_t mtime() const override { return 0; } + size_t read_count() const { return _read_count; } private: std::vector _data; + size_t _read_count = 0; }; class NativeDecoderMemoryFileReader final : public io::FileReader { @@ -419,10 +526,12 @@ class NativeDecoderMemoryFileReader final : public io::FileReader { size_t size() const override { return _data.size(); } bool closed() const override { return _closed; } int64_t mtime() const override { return 1; } + size_t read_count() const { return _read_count; } protected: Status read_at_impl(size_t offset, Slice result, size_t* bytes_read, const io::IOContext*) override { + ++_read_count; if (offset > _data.size() || result.size > _data.size() - offset) { return Status::IOError("native decoder memory read exceeds file"); } @@ -435,6 +544,7 @@ class NativeDecoderMemoryFileReader final : public io::FileReader { std::vector _data; io::Path _path; bool _closed = false; + size_t _read_count = 0; }; std::shared_ptr<::parquet::ColumnDescriptor> descriptor(::parquet::Type::type physical_type) { @@ -486,6 +596,53 @@ std::vector serialize_page(tparquet::PageHeader header, return bytes; } +std::vector serialize_plain_int32_page(const std::vector& values) { + tparquet::PageHeader header; + header.type = tparquet::PageType::DATA_PAGE; + header.__set_compressed_page_size(values.size() * sizeof(int32_t)); + header.__set_uncompressed_page_size(values.size() * sizeof(int32_t)); + header.__isset.data_page_header = true; + header.data_page_header.__set_num_values(values.size()); + header.data_page_header.__set_encoding(tparquet::Encoding::PLAIN); + header.data_page_header.__set_definition_level_encoding(tparquet::Encoding::RLE); + header.data_page_header.__set_repetition_level_encoding(tparquet::Encoding::RLE); + const auto* value_bytes = reinterpret_cast(values.data()); + return serialize_page( + header, std::vector(value_bytes, value_bytes + header.compressed_page_size)); +} + +TEST(ParquetV2NativeDecoderTest, ColumnChunkInitDoesNotReadFirstDataPage) { + tparquet::PageHeader header; + header.type = tparquet::PageType::DATA_PAGE; + header.__set_compressed_page_size(sizeof(int32_t)); + header.__set_uncompressed_page_size(sizeof(int32_t)); + header.__isset.data_page_header = true; + header.data_page_header.__set_num_values(1); + header.data_page_header.__set_encoding(tparquet::Encoding::PLAIN); + header.data_page_header.__set_definition_level_encoding(tparquet::Encoding::RLE); + header.data_page_header.__set_repetition_level_encoding(tparquet::Encoding::RLE); + auto bytes = serialize_page(header, std::vector(sizeof(int32_t))); + MemoryBufferedReader stream(bytes); + + tparquet::ColumnChunk chunk; + chunk.meta_data.__set_type(tparquet::Type::INT32); + chunk.meta_data.__set_codec(tparquet::CompressionCodec::UNCOMPRESSED); + chunk.meta_data.__set_num_values(1); + chunk.meta_data.__set_total_compressed_size(bytes.size()); + chunk.meta_data.__set_data_page_offset(0); + NativeFieldSchema field; + field.physical_type = tparquet::Type::INT32; + field.parquet_schema.__set_type(tparquet::Type::INT32); + field.parquet_schema.__set_repetition_type(tparquet::FieldRepetitionType::REQUIRED); + ParquetPageReadContext context(false, ""); + ColumnChunkReader reader(&stream, &chunk, &field, nullptr, 1, nullptr, context); + + ASSERT_TRUE(reader.init().ok()); + EXPECT_EQ(stream.read_count(), 0); + ASSERT_TRUE(reader.parse_page_header().ok()); + EXPECT_GT(stream.read_count(), 0); +} + Status materialize_level_only_page(bool data_page_v2, tparquet::Type::type physical_type, tparquet::Encoding::type encoding, bool all_null) { constexpr size_t VALUE_COUNT = 3; @@ -540,6 +697,7 @@ Status materialize_level_only_page(bool data_page_v2, tparquet::Type::type physi ColumnChunkReader chunk_reader(&reader, &chunk, &field, nullptr, VALUE_COUNT, nullptr, page_context); RETURN_IF_ERROR(chunk_reader.init()); + RETURN_IF_ERROR(chunk_reader.parse_page_header()); const auto load_status = chunk_reader.load_page_data(); EXPECT_TRUE(load_status.ok()) << load_status; RETURN_IF_ERROR(load_status); @@ -616,6 +774,7 @@ Status load_scripted_page(tparquet::PageHeader header, const std::vector chunk_reader(&reader, &chunk, &field, nullptr, 1, nullptr, context); RETURN_IF_ERROR(chunk_reader.init()); + RETURN_IF_ERROR(chunk_reader.parse_page_header()); return chunk_reader.load_page_data(); } @@ -638,6 +797,7 @@ Status load_malformed_nested_page(tparquet::PageHeader header, const std::vector ColumnChunkReader chunk_reader(&reader, &chunk, &field, nullptr, 1, nullptr, context); RETURN_IF_ERROR(chunk_reader.init()); + RETURN_IF_ERROR(chunk_reader.parse_page_header()); RETURN_IF_ERROR(chunk_reader.load_page_data()); std::vector rep_levels; size_t result_rows = 0; @@ -673,6 +833,7 @@ Status materialize_plain_int96(const std::vector& values, ColumnChunkReader chunk_reader(&reader, &chunk, &field, nullptr, values.size(), nullptr, page_context); RETURN_IF_ERROR(chunk_reader.init()); + RETURN_IF_ERROR(chunk_reader.parse_page_header()); RETURN_IF_ERROR(chunk_reader.load_page_data()); DataTypeDateTimeV2 type(6); @@ -729,6 +890,7 @@ Status materialize_selected_plain_fixed( ColumnChunkReader chunk_reader(&reader, &chunk, &field, nullptr, logical_values, nullptr, page_context); RETURN_IF_ERROR(chunk_reader.init()); + RETURN_IF_ERROR(chunk_reader.parse_page_header()); RETURN_IF_ERROR(chunk_reader.load_page_data()); ParquetDecodeContext decode_context; @@ -843,6 +1005,7 @@ Status materialize_selected_dictionary_fixed( ColumnChunkReader chunk_reader(&reader, &chunk, &field, nullptr, logical_values, nullptr, page_context); RETURN_IF_ERROR(chunk_reader.init()); + RETURN_IF_ERROR(chunk_reader.parse_page_header()); RETURN_IF_ERROR(chunk_reader.load_page_data()); ParquetDecodeContext decode_context; @@ -945,6 +1108,7 @@ Status materialize_selected_dictionary_strings(const std::vector& d ColumnChunkReader chunk_reader(&reader, &chunk, &field, nullptr, logical_values, nullptr, page_context); RETURN_IF_ERROR(chunk_reader.init()); + RETURN_IF_ERROR(chunk_reader.parse_page_header()); RETURN_IF_ERROR(chunk_reader.load_page_data()); DataTypeString type; @@ -992,6 +1156,7 @@ TEST(ParquetV2NativeDecoderTest, RawExprMapsNullableSparseRowsDirectly) { ColumnChunkReader chunk_reader(&reader, &chunk, &field, nullptr, LOGICAL_VALUES, nullptr, page_context); ASSERT_TRUE(chunk_reader.init().ok()); + ASSERT_TRUE(chunk_reader.parse_page_header().ok()); ASSERT_TRUE(chunk_reader.load_page_data().ok()); const std::vector null_runs {1, 1, 2, 1, 2}; @@ -1269,12 +1434,14 @@ TEST(ParquetV2NativeDecoderTest, DictionaryMaterializationUsesCacheAwareExecutio ParquetMaterializationState state; state.typed_dictionary = ColumnInt32::create(); assert_cast(*state.typed_dictionary).get_data() = {10, 20, 30, 40}; - ScriptedDictionaryMaterializationSource source({3, 0, 2, 1, 3}, prefer_indices); + const std::vector ids {3, 0, 2, 1, 3, 1, 0, 2, 3, 3, 1}; + ScriptedDictionaryMaterializationSource source(ids, prefer_indices); auto output = ColumnInt32::create(); - const auto status = state.materialize_dictionary(*output, source, 5); + const auto status = state.materialize_dictionary(*output, source, ids.size()); EXPECT_TRUE(status.ok()) << status; - EXPECT_EQ(output->get_data(), (ColumnInt32::Container {40, 10, 30, 20, 40})); + EXPECT_EQ(output->get_data(), + (ColumnInt32::Container {40, 10, 30, 20, 40, 20, 10, 30, 40, 40, 20})); EXPECT_EQ(state.dictionary_materialization_strategy, expected_strategy); EXPECT_EQ(source.direct_batches, expected_direct_batches); EXPECT_EQ(source.index_batches, expected_index_batches); @@ -1338,8 +1505,10 @@ TEST(ParquetV2NativeDecoderTest, DictionaryProbeMaterializesTypedValuesOnlyOnce) ScalarColumnReader reader(row_ranges, 2, chunk, nullptr, nullptr, nullptr); ASSERT_TRUE(reader.init(file, &field, bytes.size(), nullptr, "", ParquetReaderCompat {}, true) .ok()); + EXPECT_EQ(file->read_count(), 0); auto dictionary_result = reader.dictionary_values(field.data_type); ASSERT_TRUE(dictionary_result.has_value()) << dictionary_result.error(); + EXPECT_GT(file->read_count(), 0); EXPECT_EQ(reader.dictionary_materialization_count_for_test(), 1); FilterMap filter; @@ -1356,44 +1525,134 @@ TEST(ParquetV2NativeDecoderTest, DictionaryProbeMaterializesTypedValuesOnlyOnce) EXPECT_EQ(reader.dictionary_materialization_count_for_test(), 1); EXPECT_EQ(assert_cast(**matched_values).get_data(), (ColumnInt32::Container {20, 10})); + + auto nullable_output = make_nullable(std::make_shared())->create_column(); + ASSERT_TRUE(reader.append_dictionary_values(&assert_cast(*ids), + field.data_type, nullable_output.get()) + .ok()); + const auto& nullable = assert_cast(*nullable_output); + EXPECT_EQ(assert_cast(nullable.get_nested_column()).get_data(), + (ColumnInt32::Container {20, 10})); + EXPECT_EQ(nullable.get_null_map_data(), (NullMap {0, 0})); + EXPECT_EQ(reader.dictionary_materialization_count_for_test(), 1); +} + +TEST(ParquetV2NativeDecoderTest, IndexedSkipLoadsDictionaryBeforeJumpingToDataPage) { + const std::array dictionary {10, 20}; + std::vector dictionary_payload(sizeof(dictionary)); + memcpy(dictionary_payload.data(), dictionary.data(), dictionary_payload.size()); + tparquet::PageHeader dictionary_header; + dictionary_header.type = tparquet::PageType::DICTIONARY_PAGE; + dictionary_header.__set_compressed_page_size(dictionary_payload.size()); + dictionary_header.__set_uncompressed_page_size(dictionary_payload.size()); + dictionary_header.__isset.dictionary_page_header = true; + dictionary_header.dictionary_page_header.__set_num_values(dictionary.size()); + dictionary_header.dictionary_page_header.__set_encoding(tparquet::Encoding::PLAIN); + + std::vector bytes(1, 0); + const auto dictionary_page = serialize_page(dictionary_header, dictionary_payload); + bytes.insert(bytes.end(), dictionary_page.begin(), dictionary_page.end()); + + auto make_data_page = [] { + tparquet::PageHeader header; + header.type = tparquet::PageType::DATA_PAGE; + header.__set_compressed_page_size(1); + header.__set_uncompressed_page_size(1); + header.__isset.data_page_header = true; + header.data_page_header.__set_num_values(1); + header.data_page_header.__set_encoding(tparquet::Encoding::RLE_DICTIONARY); + header.data_page_header.__set_definition_level_encoding(tparquet::Encoding::RLE); + header.data_page_header.__set_repetition_level_encoding(tparquet::Encoding::RLE); + return serialize_page(header, {0}); + }; + const size_t first_data_offset = bytes.size(); + const auto first_data_page = make_data_page(); + bytes.insert(bytes.end(), first_data_page.begin(), first_data_page.end()); + const size_t second_data_offset = bytes.size(); + const auto second_data_page = make_data_page(); + bytes.insert(bytes.end(), second_data_page.begin(), second_data_page.end()); + + tparquet::OffsetIndex offset_index; + tparquet::PageLocation first_location; + first_location.__set_offset(first_data_offset); + first_location.__set_compressed_page_size(first_data_page.size()); + first_location.__set_first_row_index(0); + tparquet::PageLocation second_location; + second_location.__set_offset(second_data_offset); + second_location.__set_compressed_page_size(second_data_page.size()); + second_location.__set_first_row_index(1); + offset_index.__set_page_locations({first_location, second_location}); + + tparquet::ColumnChunk chunk; + chunk.meta_data.__set_type(tparquet::Type::INT32); + chunk.meta_data.__set_codec(tparquet::CompressionCodec::UNCOMPRESSED); + chunk.meta_data.__set_num_values(2); + chunk.meta_data.__set_total_compressed_size(bytes.size() - 1); + chunk.meta_data.__set_dictionary_page_offset(1); + chunk.meta_data.__set_data_page_offset(first_data_offset); + NativeFieldSchema field; + field.physical_type = tparquet::Type::INT32; + field.parquet_schema.__set_type(tparquet::Type::INT32); + field.parquet_schema.__set_repetition_type(tparquet::FieldRepetitionType::REQUIRED); + MemoryBufferedReader stream(bytes); + ParquetPageReadContext context(false, ""); + ColumnChunkReader reader(&stream, &chunk, &field, &offset_index, 2, nullptr, + context); + + ASSERT_TRUE(reader.init().ok()); + EXPECT_EQ(stream.read_count(), 0); + ASSERT_TRUE(reader.next_page().ok()); + EXPECT_GT(stream.read_count(), 0); + bool has_dict = false; + const size_t reads_after_seek = stream.read_count(); + ASSERT_TRUE(reader.load_dictionary_page(&has_dict).ok()); + EXPECT_TRUE(has_dict); + EXPECT_EQ(stream.read_count(), reads_after_seek); + ASSERT_NE(reader.dictionary_decoder(), nullptr); + EXPECT_EQ(reader.dictionary_decoder()->dictionary_size(), dictionary.size()); + ASSERT_TRUE(reader.parse_page_header().ok()); + EXPECT_EQ(reader.page_start_row(), 1); } TEST(ParquetV2NativeDecoderTest, DictionaryRepeatedRunsGatherDirectlyIntoDestination) { - const std::array dictionary_values {10, 20}; - auto dictionary = make_unique_buffer(sizeof(dictionary_values)); - memcpy(dictionary.get(), dictionary_values.data(), sizeof(dictionary_values)); - std::unique_ptr decoder; - ASSERT_TRUE( - Decoder::get_decoder(tparquet::Type::INT32, tparquet::Encoding::RLE_DICTIONARY, decoder) - .ok()); - decoder->set_type_length(sizeof(int32_t)); - ASSERT_TRUE(decoder->set_dict(dictionary, sizeof(dictionary_values), dictionary_values.size()) + for (const auto encoding : + {tparquet::Encoding::RLE_DICTIONARY, tparquet::Encoding::PLAIN_DICTIONARY}) { + const std::array dictionary_values {10, 20}; + auto dictionary = make_unique_buffer(sizeof(dictionary_values)); + memcpy(dictionary.get(), dictionary_values.data(), sizeof(dictionary_values)); + std::unique_ptr decoder; + ASSERT_TRUE(Decoder::get_decoder(tparquet::Type::INT32, encoding, decoder).ok()); + decoder->set_type_length(sizeof(int32_t)); + ASSERT_TRUE( + decoder->set_dict(dictionary, sizeof(dictionary_values), dictionary_values.size()) .ok()); - faststring encoded_ids; - RleEncoder encoder(&encoded_ids, 1); - for (size_t row = 0; row < 64; ++row) { - encoder.Put(1); - } - encoder.Flush(); - std::vector payload(encoded_ids.size() + 1); - payload[0] = 1; - memcpy(payload.data() + 1, encoded_ids.data(), encoded_ids.size()); - Slice id_slice(payload.data(), payload.size()); - ASSERT_TRUE(decoder->set_data(&id_slice).ok()); - - DataTypeInt32 type; - auto output = type.create_column(); - ParquetMaterializationState state; - ParquetDecodeContext context {.physical_type = ParquetPhysicalType::INT32, - .encoding = ParquetValueEncoding::DICTIONARY}; - ASSERT_TRUE( - type.get_serde()->read_column_from_parquet(*output, *decoder, context, 64, state).ok()); - EXPECT_EQ(state.dictionary_materialization_strategy, - ParquetDictionaryMaterializationStrategy::DIRECT); - ASSERT_EQ(output->size(), 64); - for (size_t row = 0; row < output->size(); ++row) { - EXPECT_EQ(assert_cast(*output).get_element(row), 20); + faststring encoded_ids; + RleEncoder encoder(&encoded_ids, 1); + for (size_t row = 0; row < 64; ++row) { + encoder.Put(1); + } + encoder.Flush(); + std::vector payload(encoded_ids.size() + 1); + payload[0] = 1; + memcpy(payload.data() + 1, encoded_ids.data(), encoded_ids.size()); + Slice id_slice(payload.data(), payload.size()); + ASSERT_TRUE(decoder->set_data(&id_slice).ok()); + + DataTypeInt32 type; + auto output = type.create_column(); + ParquetMaterializationState state; + ParquetDecodeContext context {.physical_type = ParquetPhysicalType::INT32, + .encoding = ParquetValueEncoding::DICTIONARY}; + ASSERT_TRUE(type.get_serde() + ->read_column_from_parquet(*output, *decoder, context, 64, state) + .ok()); + EXPECT_EQ(state.dictionary_materialization_strategy, + ParquetDictionaryMaterializationStrategy::DIRECT); + ASSERT_EQ(output->size(), 64); + for (size_t row = 0; row < output->size(); ++row) { + EXPECT_EQ(assert_cast(*output).get_element(row), 20); + } } } @@ -2844,6 +3103,7 @@ TEST(ParquetV2NativeDecoderTest, DecompressionScratchStaysActiveUntilPageExhaust static_cast(LARGE_VALUE_COUNT) + 1, nullptr, context); ASSERT_TRUE(reader.init().ok()); + ASSERT_TRUE(reader.parse_page_header().ok()); ASSERT_TRUE(reader.load_page_data().ok()); ASSERT_GT(reader.active_decoder_scratch_bytes(), 1UL << 20); ASSERT_TRUE(reader.skip_values(LARGE_VALUE_COUNT).ok()); @@ -3067,6 +3327,218 @@ TEST(ParquetV2NativeDecoderTest, ShiftedOffsetIndexFallsBackToSequentialPages) { verify_fallback(true); } +TEST(ParquetV2NativeDecoderTest, LazyFlatIndexedSkipValidatesFirstPageCardinality) { + const auto first_page = serialize_plain_int32_page({10}); + const auto second_page = serialize_plain_int32_page({20, 21}); + const auto third_page = serialize_plain_int32_page({30}); + const auto fourth_page = serialize_plain_int32_page({40, 41}); + std::vector bytes = first_page; + const size_t second_offset = bytes.size(); + bytes.insert(bytes.end(), second_page.begin(), second_page.end()); + const size_t third_offset = bytes.size(); + bytes.insert(bytes.end(), third_page.begin(), third_page.end()); + const size_t fourth_offset = bytes.size(); + bytes.insert(bytes.end(), fourth_page.begin(), fourth_page.end()); + + tparquet::OffsetIndex offset_index; + std::vector locations(4); + locations[0].__set_offset(0); + locations[0].__set_compressed_page_size(first_page.size()); + locations[0].__set_first_row_index(0); + locations[1].__set_offset(second_offset); + locations[1].__set_compressed_page_size(second_page.size()); + locations[1].__set_first_row_index(2); + locations[2].__set_offset(third_offset); + locations[2].__set_compressed_page_size(third_page.size()); + locations[2].__set_first_row_index(4); + locations[3].__set_offset(fourth_offset); + locations[3].__set_compressed_page_size(fourth_page.size()); + locations[3].__set_first_row_index(5); + offset_index.__set_page_locations(locations); + + tparquet::ColumnChunk chunk; + chunk.meta_data.__set_type(tparquet::Type::INT32); + chunk.meta_data.__set_codec(tparquet::CompressionCodec::UNCOMPRESSED); + chunk.meta_data.__set_num_values(6); + chunk.meta_data.__set_total_compressed_size(bytes.size()); + chunk.meta_data.__set_data_page_offset(0); + NativeFieldSchema field; + field.physical_type = tparquet::Type::INT32; + field.data_type = std::make_shared(); + field.parquet_schema.__set_type(tparquet::Type::INT32); + field.parquet_schema.__set_repetition_type(tparquet::FieldRepetitionType::REQUIRED); + auto file = std::make_shared(bytes); + const auto row_ranges = ::doris::RowRanges::create_single(2, 4); + ScalarColumnReader reader(row_ranges, 6, chunk, &offset_index, nullptr, nullptr); + ASSERT_TRUE(reader.init(file, &field, bytes.size(), nullptr, "", ParquetReaderCompat {}, true) + .ok()); + + FilterMap filter; + ASSERT_TRUE(filter.init(nullptr, 2, false).ok()); + ColumnPtr values = ColumnInt32::create(); + size_t rows = 0; + bool eof = false; + const auto status = reader.read_column_data(values, field.data_type, nullptr, filter, 2, &rows, + &eof, false); + EXPECT_TRUE(status.is()) << status; +} + +TEST(ParquetV2NativeDecoderTest, LazyFixedWidthFilterUsesReconciledFirstPageRange) { + const auto first_page = serialize_plain_int32_page({10, 11}); + const auto second_page = serialize_plain_int32_page({20}); + std::vector bytes = first_page; + bytes.insert(bytes.end(), second_page.begin(), second_page.end()); + bytes.push_back(0); + + tparquet::OffsetIndex offset_index; + tparquet::PageLocation first_location; + first_location.__set_offset(0); + first_location.__set_compressed_page_size(first_page.size() + 1); + first_location.__set_first_row_index(0); + tparquet::PageLocation second_location; + second_location.__set_offset(first_page.size() + 1); + second_location.__set_compressed_page_size(second_page.size()); + second_location.__set_first_row_index(1); + offset_index.__set_page_locations({first_location, second_location}); + + tparquet::ColumnChunk chunk; + chunk.meta_data.__set_type(tparquet::Type::INT32); + chunk.meta_data.__set_codec(tparquet::CompressionCodec::UNCOMPRESSED); + chunk.meta_data.__set_num_values(3); + chunk.meta_data.__set_total_compressed_size(bytes.size()); + chunk.meta_data.__set_data_page_offset(0); + chunk.meta_data.__set_encodings({tparquet::Encoding::PLAIN}); + NativeFieldSchema field; + field.physical_type = tparquet::Type::INT32; + field.data_type = std::make_shared(); + field.parquet_schema.__set_type(tparquet::Type::INT32); + field.parquet_schema.__set_repetition_type(tparquet::FieldRepetitionType::REQUIRED); + auto file = std::make_shared(bytes); + const auto row_ranges = ::doris::RowRanges::create_single(0, 2); + ScalarColumnReader reader(row_ranges, 3, chunk, &offset_index, nullptr, nullptr); + ASSERT_TRUE(reader.init(file, &field, bytes.size(), nullptr, "", ParquetReaderCompat {}, true) + .ok()); + + FilterMap filter; + ASSERT_TRUE(filter.init(nullptr, 2, false).ok()); + IColumn::Filter row_filter; + size_t rows = 0; + bool eof = false; + bool used_filter = false; + const auto status = reader.read_fixed_width_filter( + {create_int32_raw_comparison(0, "ge", TExprOpcode::GE, 0)}, 0, filter, 2, nullptr, + &row_filter, &rows, &eof, &used_filter); + ASSERT_TRUE(status.ok()) << status; + EXPECT_TRUE(used_filter); + EXPECT_EQ(rows, 2); + EXPECT_EQ(row_filter, (IColumn::Filter {1, 1})); +} + +TEST(ParquetV2NativeDecoderTest, LazyFlatIndexedFallbackUsesReconciledFirstPageRange) { + const auto first_page = serialize_plain_int32_page({10, 11}); + const auto second_page = serialize_plain_int32_page({20}); + std::vector bytes = first_page; + bytes.insert(bytes.end(), second_page.begin(), second_page.end()); + bytes.push_back(0); + + tparquet::OffsetIndex offset_index; + tparquet::PageLocation first_location; + first_location.__set_offset(0); + first_location.__set_compressed_page_size(first_page.size() + 1); + first_location.__set_first_row_index(0); + tparquet::PageLocation second_location; + second_location.__set_offset(first_page.size() + 1); + second_location.__set_compressed_page_size(second_page.size()); + second_location.__set_first_row_index(1); + offset_index.__set_page_locations({first_location, second_location}); + + tparquet::ColumnChunk chunk; + chunk.meta_data.__set_type(tparquet::Type::INT32); + chunk.meta_data.__set_codec(tparquet::CompressionCodec::UNCOMPRESSED); + chunk.meta_data.__set_num_values(3); + chunk.meta_data.__set_total_compressed_size(bytes.size()); + chunk.meta_data.__set_data_page_offset(0); + NativeFieldSchema field; + field.physical_type = tparquet::Type::INT32; + field.data_type = std::make_shared(); + field.parquet_schema.__set_type(tparquet::Type::INT32); + field.parquet_schema.__set_repetition_type(tparquet::FieldRepetitionType::REQUIRED); + auto file = std::make_shared(bytes); + const auto row_ranges = ::doris::RowRanges::create_single(0, 2); + ScalarColumnReader reader(row_ranges, 3, chunk, &offset_index, nullptr, nullptr); + ASSERT_TRUE(reader.init(file, &field, bytes.size(), nullptr, "", ParquetReaderCompat {}, true) + .ok()); + + FilterMap filter; + ASSERT_TRUE(filter.init(nullptr, 2, false).ok()); + ColumnPtr values = ColumnInt32::create(); + size_t rows = 0; + bool eof = false; + ASSERT_TRUE( + reader.read_column_data(values, field.data_type, nullptr, filter, 2, &rows, &eof, false) + .ok()); + ASSERT_EQ(rows, 2); + EXPECT_EQ(assert_cast(*values).get_data(), + (ColumnInt32::Container {10, 11})); +} + +TEST(ParquetV2NativeDecoderTest, LazyNestedV2SeekValidatesFirstPageRowRange) { + auto make_page = [](int32_t value) { + const std::vector repetition_levels {2, 0}; + std::vector payload = repetition_levels; + const auto* value_bytes = reinterpret_cast(&value); + payload.insert(payload.end(), value_bytes, value_bytes + sizeof(value)); + tparquet::PageHeader header; + header.type = tparquet::PageType::DATA_PAGE_V2; + header.__set_compressed_page_size(payload.size()); + header.__set_uncompressed_page_size(payload.size()); + header.__isset.data_page_header_v2 = true; + header.data_page_header_v2.__set_num_values(1); + header.data_page_header_v2.__set_num_rows(1); + header.data_page_header_v2.__set_num_nulls(0); + header.data_page_header_v2.__set_encoding(tparquet::Encoding::PLAIN); + header.data_page_header_v2.__set_repetition_levels_byte_length(repetition_levels.size()); + header.data_page_header_v2.__set_definition_levels_byte_length(0); + header.data_page_header_v2.__set_is_compressed(false); + return serialize_page(header, payload); + }; + const auto first_page = make_page(10); + const auto second_page = make_page(20); + std::vector bytes = first_page; + const size_t second_offset = bytes.size(); + bytes.insert(bytes.end(), second_page.begin(), second_page.end()); + + tparquet::OffsetIndex offset_index; + tparquet::PageLocation first_location; + first_location.__set_offset(0); + first_location.__set_compressed_page_size(first_page.size()); + first_location.__set_first_row_index(0); + tparquet::PageLocation second_location; + second_location.__set_offset(second_offset); + second_location.__set_compressed_page_size(second_page.size()); + second_location.__set_first_row_index(2); + offset_index.__set_page_locations({first_location, second_location}); + + tparquet::ColumnChunk chunk; + chunk.meta_data.__set_type(tparquet::Type::INT32); + chunk.meta_data.__set_codec(tparquet::CompressionCodec::UNCOMPRESSED); + chunk.meta_data.__set_num_values(2); + chunk.meta_data.__set_total_compressed_size(bytes.size()); + chunk.meta_data.__set_data_page_offset(0); + NativeFieldSchema field; + field.physical_type = tparquet::Type::INT32; + field.repetition_level = 1; + ParquetPageReadContext context(false, ""); + MemoryBufferedReader stream(bytes); + ColumnChunkReader reader(&stream, &chunk, &field, &offset_index, 3, nullptr, + context); + + ASSERT_TRUE(reader.init().ok()); + EXPECT_EQ(stream.read_count(), 0); + const auto status = reader.seek_to_nested_row(2); + EXPECT_TRUE(status.is()) << status; +} + TEST(ParquetV2NativeDecoderTest, FlatPagesRejectLogicalAndPhysicalCardinalityMismatch) { auto init_chunk = [](tparquet::PageHeader header, bool with_offset_index) { std::vector payload(static_cast(header.compressed_page_size), 0); @@ -3092,11 +3564,13 @@ TEST(ParquetV2NativeDecoderTest, FlatPagesRejectLogicalAndPhysicalCardinalityMis if (with_offset_index) { ColumnChunkReader reader_with_index(&reader, &chunk, &field, &offset_index, 1, nullptr, context); - return reader_with_index.init(); + RETURN_IF_ERROR(reader_with_index.init()); + return reader_with_index.parse_page_header(); } ColumnChunkReader sequential_reader(&reader, &chunk, &field, nullptr, 1, nullptr, context); - return sequential_reader.init(); + RETURN_IF_ERROR(sequential_reader.init()); + return sequential_reader.parse_page_header(); }; tparquet::PageHeader v2; @@ -3165,7 +3639,8 @@ TEST(ParquetV2NativeDecoderTest, NestedV2PageRejectsOffsetIndexRowSpanMismatch) ColumnChunkReader reader(&stream, &chunk, &field, &offset_index, /*total_rows=*/2, nullptr, context); - const auto status = reader.init(); + ASSERT_TRUE(reader.init().ok()); + const auto status = reader.parse_page_header(); EXPECT_TRUE(status.is()) << status; } @@ -3374,6 +3849,7 @@ TEST(ParquetV2NativeDecoderTest, NestedV1ContinuationRemainsValidAfterFirstRowSt ColumnChunkReader chunk_reader(&reader, &chunk, &field, nullptr, 1, nullptr, context); ASSERT_TRUE(chunk_reader.init().ok()); + ASSERT_TRUE(chunk_reader.parse_page_header().ok()); ASSERT_TRUE(chunk_reader.load_page_data().ok()); std::vector levels; size_t rows = 0; @@ -3429,6 +3905,7 @@ TEST(ParquetV2NativeDecoderTest, NestedV1IgnoresUnverifiableOffsetIndexRows) { ColumnChunkReader chunk_reader(&stream, &chunk, &field, &offset_index, 1, nullptr, context); ASSERT_TRUE(chunk_reader.init().ok()); + ASSERT_TRUE(chunk_reader.parse_page_header().ok()); ASSERT_TRUE(chunk_reader.load_page_data().ok()); std::vector levels; size_t rows = 0; @@ -3440,6 +3917,141 @@ TEST(ParquetV2NativeDecoderTest, NestedV1IgnoresUnverifiableOffsetIndexRows) { EXPECT_EQ(levels, std::vector({0, 1, 1})); } +TEST(ParquetV2NativeDecoderTest, LazyNestedV1SeekDoesNotOutrunPhysicalPages) { + auto make_page = [] { + tparquet::PageHeader header; + header.type = tparquet::PageType::DATA_PAGE; + const std::vector payload {2, 0, 0, 0, 2, 0, 0, 0, 0, 0}; + header.__set_compressed_page_size(payload.size()); + header.__set_uncompressed_page_size(payload.size()); + header.__isset.data_page_header = true; + header.data_page_header.__set_num_values(1); + header.data_page_header.__set_encoding(tparquet::Encoding::PLAIN); + header.data_page_header.__set_repetition_level_encoding(tparquet::Encoding::RLE); + header.data_page_header.__set_definition_level_encoding(tparquet::Encoding::RLE); + return serialize_page(header, payload); + }; + const auto first_page = make_page(); + const auto second_page = make_page(); + std::vector bytes = first_page; + bytes.insert(bytes.end(), second_page.begin(), second_page.end()); + + MemoryBufferedReader stream(bytes); + tparquet::ColumnChunk chunk; + chunk.meta_data.__set_type(tparquet::Type::INT32); + chunk.meta_data.__set_codec(tparquet::CompressionCodec::UNCOMPRESSED); + chunk.meta_data.__set_num_values(2); + chunk.meta_data.__set_total_compressed_size(bytes.size()); + chunk.meta_data.__set_data_page_offset(0); + NativeFieldSchema field; + field.physical_type = tparquet::Type::INT32; + field.repetition_level = 1; + tparquet::OffsetIndex offset_index; + tparquet::PageLocation first_location; + first_location.__set_offset(0); + first_location.__set_compressed_page_size(first_page.size()); + first_location.__set_first_row_index(0); + tparquet::PageLocation second_location; + second_location.__set_offset(first_page.size()); + second_location.__set_compressed_page_size(second_page.size()); + second_location.__set_first_row_index(1); + offset_index.__set_page_locations({first_location, second_location}); + ParquetPageReadContext context(false, ""); + ColumnChunkReader chunk_reader(&stream, &chunk, &field, &offset_index, 2, nullptr, + context); + + ASSERT_TRUE(chunk_reader.init().ok()); + EXPECT_EQ(stream.read_count(), 0); + ASSERT_TRUE(chunk_reader.seek_to_nested_row(1).ok()); + std::vector levels; + size_t rows = 0; + bool cross_page = false; + ASSERT_TRUE(chunk_reader.load_page_nested_rows(levels, 1, &rows, &cross_page).ok()); + if (cross_page) { + const auto status = chunk_reader.load_cross_page_nested_row(levels, &cross_page); + ASSERT_TRUE(status.ok()) << status; + } + EXPECT_EQ(rows, 1); + EXPECT_FALSE(cross_page); +} + +TEST(ParquetV2NativeDecoderTest, LazyDictionaryNestedV1SeekChecksFirstDataPage) { + tparquet::PageHeader dictionary_header; + dictionary_header.type = tparquet::PageType::DICTIONARY_PAGE; + dictionary_header.__set_compressed_page_size(sizeof(int32_t)); + dictionary_header.__set_uncompressed_page_size(sizeof(int32_t)); + dictionary_header.__isset.dictionary_page_header = true; + dictionary_header.dictionary_page_header.__set_num_values(1); + dictionary_header.dictionary_page_header.__set_encoding(tparquet::Encoding::PLAIN); + const int32_t dictionary_value = 7; + const auto* dictionary_bytes = reinterpret_cast(&dictionary_value); + auto bytes = serialize_page( + dictionary_header, + std::vector(dictionary_bytes, dictionary_bytes + sizeof(dictionary_value))); + + auto make_data_page = [] { + tparquet::PageHeader header; + header.type = tparquet::PageType::DATA_PAGE; + const std::vector payload {2, 0, 0, 0, 2, 0, 0, 0, 0, 0}; + header.__set_compressed_page_size(payload.size()); + header.__set_uncompressed_page_size(payload.size()); + header.__isset.data_page_header = true; + header.data_page_header.__set_num_values(1); + header.data_page_header.__set_encoding(tparquet::Encoding::PLAIN); + header.data_page_header.__set_repetition_level_encoding(tparquet::Encoding::RLE); + header.data_page_header.__set_definition_level_encoding(tparquet::Encoding::RLE); + return serialize_page(header, payload); + }; + const auto first_page = make_data_page(); + const size_t first_page_offset = bytes.size(); + bytes.insert(bytes.end(), first_page.begin(), first_page.end()); + const auto second_page = make_data_page(); + const size_t second_page_offset = bytes.size(); + bytes.insert(bytes.end(), second_page.begin(), second_page.end()); + const size_t chunk_size = bytes.size(); + bytes.resize(chunk_size + 16, 0); + + MemoryBufferedReader stream(bytes); + tparquet::ColumnChunk chunk; + chunk.meta_data.__set_type(tparquet::Type::INT32); + chunk.meta_data.__set_codec(tparquet::CompressionCodec::UNCOMPRESSED); + chunk.meta_data.__set_num_values(2); + chunk.meta_data.__set_total_compressed_size(chunk_size); + chunk.meta_data.__set_dictionary_page_offset(0); + chunk.meta_data.__set_data_page_offset(first_page_offset); + NativeFieldSchema field; + field.physical_type = tparquet::Type::INT32; + field.repetition_level = 1; + tparquet::OffsetIndex offset_index; + tparquet::PageLocation first_location; + first_location.__set_offset(first_page_offset); + first_location.__set_compressed_page_size(first_page.size()); + first_location.__set_first_row_index(0); + tparquet::PageLocation second_location; + second_location.__set_offset(second_page_offset); + second_location.__set_compressed_page_size(second_page.size()); + second_location.__set_first_row_index(1); + offset_index.__set_page_locations({first_location, second_location}); + ColumnChunkRange padded_range {.offset = 0, .length = bytes.size()}; + ParquetPageReadContext context(false, ""); + ColumnChunkReader chunk_reader(&stream, &chunk, &field, &offset_index, 2, nullptr, + context, &padded_range); + + ASSERT_TRUE(chunk_reader.init().ok()); + EXPECT_EQ(stream.read_count(), 0); + ASSERT_TRUE(chunk_reader.seek_to_nested_row(1).ok()); + std::vector levels; + size_t rows = 0; + bool cross_page = false; + ASSERT_TRUE(chunk_reader.load_page_nested_rows(levels, 1, &rows, &cross_page).ok()); + if (cross_page) { + const auto status = chunk_reader.load_cross_page_nested_row(levels, &cross_page); + ASSERT_TRUE(status.ok()) << status; + } + EXPECT_EQ(rows, 1); + EXPECT_FALSE(cross_page); +} + TEST(ParquetV2NativeDecoderTest, NestedV1DiscardedOffsetIndexStopsAtLogicalChunkEnd) { tparquet::PageHeader header; header.type = tparquet::PageType::DATA_PAGE; @@ -3477,6 +4089,7 @@ TEST(ParquetV2NativeDecoderTest, NestedV1DiscardedOffsetIndexStopsAtLogicalChunk ColumnChunkReader chunk_reader(&stream, &chunk, &field, &offset_index, 1, nullptr, context, &padded_range); ASSERT_TRUE(chunk_reader.init().ok()); + ASSERT_TRUE(chunk_reader.parse_page_header().ok()); ASSERT_TRUE(chunk_reader.load_page_data().ok()); std::vector levels; size_t rows = 0; @@ -3634,6 +4247,7 @@ TEST(ParquetV2NativeDecoderTest, OptionalV2FixedWidthPageRejectsExtentBeforeAllo ColumnChunkReader reader(&stream, &chunk, &field, nullptr, 1, nullptr, context); ASSERT_TRUE(reader.init().ok()); + ASSERT_TRUE(reader.parse_page_header().ok()); EXPECT_TRUE(reader.load_page_data().is()); EXPECT_LT(reader.retained_decoder_scratch_bytes(), 64UL << 10); } @@ -3679,6 +4293,7 @@ TEST(ParquetV2NativeDecoderTest, RepeatedV2FixedWidthPageRejectsExtentBeforeAllo ParquetPageReadContext context(false, ""); ColumnChunkReader reader(&stream, &chunk, &field, nullptr, 1, nullptr, context); ASSERT_TRUE(reader.init().ok()); + ASSERT_TRUE(reader.parse_page_header().ok()); EXPECT_TRUE(reader.load_page_data().is()); EXPECT_LT(reader.retained_decoder_scratch_bytes(), 64UL << 10); } @@ -3717,6 +4332,7 @@ TEST(ParquetV2NativeDecoderTest, VariableWidthDataPagePreflightsCompressedExtent ParquetPageReadContext context(false, ""); ColumnChunkReader reader(&stream, &chunk, &field, nullptr, 1, nullptr, context); ASSERT_TRUE(reader.init().ok()); + ASSERT_TRUE(reader.parse_page_header().ok()); EXPECT_TRUE(reader.load_page_data().is()); EXPECT_LT(reader.retained_decoder_scratch_bytes(), 64UL << 10); EXPECT_TRUE(load_scripted_page(header, payload, tparquet::CompressionCodec::SNAPPY, true, @@ -3917,6 +4533,7 @@ TEST(ParquetV2NativeDecoderTest, ColumnChunkSkipsIndexPageBeforeInitializingData context); const auto init_status = chunk_reader.init(); ASSERT_TRUE(init_status.ok()) << init_status; + ASSERT_TRUE(chunk_reader.parse_page_header().ok()); EXPECT_EQ(chunk_reader.remaining_num_values(), 1); ASSERT_TRUE(chunk_reader.load_page_data().ok()); } @@ -3958,6 +4575,7 @@ TEST(ParquetV2NativeDecoderTest, ColumnChunkSkipsUnknownAuxiliaryPage) { context); const auto init_status = chunk_reader.init(); ASSERT_TRUE(init_status.ok()) << init_status; + ASSERT_TRUE(chunk_reader.parse_page_header().ok()); EXPECT_EQ(chunk_reader.remaining_num_values(), 1); } @@ -4071,6 +4689,92 @@ TEST(ParquetV2NativeDecoderTest, FixedLengthStringsAppendAsOneContiguousSpan) { EXPECT_EQ(column.get_data_at(2).to_string_view(), "ccc"); } +TEST(ParquetV2NativeDecoderTest, DictionaryStringGatherAppendsCompactSurvivors) { + ColumnString dictionary; + dictionary.insert_data("alpha", 5); + dictionary.insert_data("bravo", 5); + dictionary.insert_data("charlie", 7); + dictionary.insert_data("delta", 5); + ColumnString destination; + destination.insert_data("prefix", 6); + const std::array indices {3, 1, 2}; + + ASSERT_TRUE(try_simd_insert_parquet_dictionary_indices(destination, dictionary, indices.data(), + indices.size())); + ASSERT_EQ(destination.size(), 4); + EXPECT_EQ(destination.get_data_at(0).to_string_view(), "prefix"); + EXPECT_EQ(destination.get_data_at(1).to_string_view(), "delta"); + EXPECT_EQ(destination.get_data_at(2).to_string_view(), "bravo"); + EXPECT_EQ(destination.get_data_at(3).to_string_view(), "charlie"); +} + +TEST(ParquetV2NativeDecoderTest, DictionaryDecimalGatherAppendsCompactSurvivors) { + const std::array indices {3, 1, 2, 0, 0, 2, 1, 3}; + const auto verify = [&]() { + using ValueType = typename ColumnType::value_type; + auto dictionary = ColumnType::create(0, 2); + auto& dictionary_data = dictionary->get_data(); + dictionary_data.resize(4); + for (size_t row = 0; row < dictionary_data.size(); ++row) { + dictionary_data[row].value = static_cast(100 * (row + 1)); + } + auto destination = ColumnType::create(0, 2); + destination->get_data().push_back(ValueType {50}); + + ASSERT_TRUE(try_simd_insert_parquet_dictionary_indices(*destination, *dictionary, + indices.data(), indices.size())); + ASSERT_EQ(destination->size(), indices.size() + 1); + EXPECT_EQ(destination->get_data()[0].value, 50); + for (size_t row = 0; row < indices.size(); ++row) { + EXPECT_EQ(destination->get_data()[row + 1].value, dictionary_data[indices[row]].value); + } + }; + verify.template operator()(); + verify.template operator()(); +} + +template +void expect_fixed_width_dictionary_gather() { + using ColumnType = ColumnVector; + using ValueType = typename ColumnType::value_type; + static_assert(sizeof(ValueType) == 4 || sizeof(ValueType) == 8); + + auto dictionary = ColumnType::create(); + auto& dictionary_data = dictionary->get_data(); + dictionary_data.resize(4); + for (size_t row = 0; row < dictionary_data.size(); ++row) { + const uint64_t bits = 0x0102030405060708ULL + row; + memcpy(&dictionary_data[row], &bits, sizeof(ValueType)); + } + auto destination = ColumnType::create(); + destination->get_data().push_back(dictionary_data[0]); + const std::array indices {3, 1, 2, 0, 0, 2, 1, 3}; + + ASSERT_TRUE(try_simd_insert_parquet_dictionary_indices(*destination, *dictionary, + indices.data(), indices.size())); + ASSERT_EQ(destination->size(), indices.size() + 1); + for (size_t row = 0; row < indices.size(); ++row) { + EXPECT_EQ(0, memcmp(&destination->get_data()[row + 1], &dictionary_data[indices[row]], + sizeof(ValueType))); + } +} + +TEST(ParquetV2NativeDecoderTest, DictionaryGatherSupportsEverySimdFixedWidthColumn) { + expect_fixed_width_dictionary_gather(); + expect_fixed_width_dictionary_gather(); + expect_fixed_width_dictionary_gather(); + expect_fixed_width_dictionary_gather(); + expect_fixed_width_dictionary_gather(); + expect_fixed_width_dictionary_gather(); + expect_fixed_width_dictionary_gather(); + expect_fixed_width_dictionary_gather(); + expect_fixed_width_dictionary_gather(); + expect_fixed_width_dictionary_gather(); + expect_fixed_width_dictionary_gather(); + expect_fixed_width_dictionary_gather(); + expect_fixed_width_dictionary_gather(); +} + TEST(ParquetV2NativeDecoderTest, ComplexPageStatisticsPreservePerLeafCrossings) { ColumnChunkReaderStatistics first_chunk; first_chunk.page_read_counter = 1; diff --git a/be/test/format_v2/parquet/parquet_benchmark_scenarios_test.cpp b/be/test/format_v2/parquet/parquet_benchmark_scenarios_test.cpp index 490a0bde204a74..89e1f56ac3e63c 100644 --- a/be/test/format_v2/parquet/parquet_benchmark_scenarios_test.cpp +++ b/be/test/format_v2/parquet/parquet_benchmark_scenarios_test.cpp @@ -21,7 +21,9 @@ #include #include +#include #include +#include #include namespace doris::parquet_benchmark { @@ -29,6 +31,7 @@ namespace { TEST(ParquetBenchmarkScenariosTest, DecoderMatrixCoversNativeEncodingAndTypeFamilies) { const auto scenarios = decoder_scenarios(); + EXPECT_EQ(scenarios.size() * 6 * 2, size_t {228}); const std::set> actual = [&] { std::set> values; for (const auto& scenario : scenarios) { @@ -61,8 +64,55 @@ TEST(ParquetBenchmarkScenariosTest, DecoderMatrixCoversNativeEncodingAndTypeFami EXPECT_EQ(actual, expected); } +TEST(ParquetBenchmarkScenariosTest, KernelMatrixCoversEverySimdStageAndBoundaryShape) { + const auto scenarios = kernel_scenarios(); + EXPECT_EQ(scenarios.size(), size_t {80}); + const std::map> expected_types { + {Kernel::BYTE_STREAM_SPLIT, {ValueType::FLOAT, ValueType::DOUBLE}}, + {Kernel::DELTA_PREFIX_SUM, {ValueType::INT32, ValueType::INT64}}, + {Kernel::DICTIONARY_GATHER, + {ValueType::INT32, ValueType::INT64, ValueType::FLOAT, ValueType::DOUBLE}}, + {Kernel::NULLABLE_EXPAND, + {ValueType::INT32, ValueType::INT64, ValueType::FLOAT, ValueType::DOUBLE}}, + {Kernel::RAW_PREDICATE, + {ValueType::INT32, ValueType::INT64, ValueType::FLOAT, ValueType::DOUBLE}}, + }; + for (const auto& [kernel, value_types] : expected_types) { + for (const auto value_type : value_types) { + EXPECT_TRUE(std::ranges::any_of(scenarios, + [&](const KernelScenario& scenario) { + return scenario.kernel == kernel && + scenario.value_type == value_type; + })) + << "missing SIMD width for kernel " << to_string(kernel); + } + } + for (const int selectivity : {0, 1, 10, 50, 90, 100}) { + EXPECT_TRUE(std::ranges::any_of(scenarios, [&](const KernelScenario& scenario) { + return scenario.kernel == Kernel::RAW_PREDICATE && + scenario.selectivity_percent == selectivity; + })) << "missing raw predicate selectivity"; + } + for (const int null_percent : {0, 1, 10, 50, 90}) { + for (const auto pattern : {Pattern::CLUSTERED, Pattern::ALTERNATING}) { + EXPECT_TRUE(std::ranges::any_of(scenarios, [&](const KernelScenario& scenario) { + return scenario.kernel == Kernel::NULLABLE_EXPAND && + scenario.null_percent == null_percent && scenario.pattern == pattern; + })) << "missing nullable expansion shape"; + } + } + for (const size_t dictionary_entries : {32, 4096, 262144}) { + EXPECT_TRUE(std::ranges::any_of(scenarios, [&](const KernelScenario& scenario) { + return scenario.kernel == Kernel::DICTIONARY_GATHER && + scenario.dictionary_entries == dictionary_entries; + })) << "missing dictionary working-set size"; + } +} + TEST(ParquetBenchmarkScenariosTest, ReaderMatrixCoversNullableSparseAndProjectionAxes) { const auto scenarios = reader_scenarios(); + // Keep the exact count aligned with the upstream complex-residual scenario retained by rebase. + EXPECT_EQ(scenarios.size(), size_t {167}); for (const int null_percent : {0, 1, 10, 50, 90}) { for (const auto pattern : {Pattern::CLUSTERED, Pattern::ALTERNATING}) { for (const int selectivity : {0, 1, 10, 50, 90, 100}) { @@ -86,7 +136,8 @@ TEST(ParquetBenchmarkScenariosTest, ReaderMatrixCoversOperationsEncodingsAndSche const auto scenarios = reader_scenarios(); for (const auto operation : {ReaderOperation::OPEN_TO_FIRST_BLOCK, ReaderOperation::FULL_SCAN, - ReaderOperation::PREDICATE_SCAN, ReaderOperation::LIMIT_1, ReaderOperation::LIMIT_1000}) { + ReaderOperation::PREDICATE_SCAN, ReaderOperation::COMPLEX_RESIDUAL_SCAN, + ReaderOperation::LIMIT_1, ReaderOperation::LIMIT_1000}) { EXPECT_TRUE(std::ranges::any_of(scenarios, [&](const ReaderScenario& scenario) { return scenario.operation == operation; })); @@ -109,6 +160,26 @@ TEST(ParquetBenchmarkScenariosTest, ReaderMatrixCoversOperationsEncodingsAndSche } } +TEST(ParquetBenchmarkScenariosTest, ReaderMatrixHasExactUniqueRegistrationNames) { + const auto scenarios = reader_scenarios(); + EXPECT_EQ(scenarios.size(), size_t {167}); + + std::set names; + for (const auto& scenario : scenarios) { + names.insert(reader_scenario_name(scenario)); + } + EXPECT_EQ(names.size(), scenarios.size()); +} + +TEST(ParquetBenchmarkScenariosTest, ReaderMatrixCoversComplexResidualLazyMaterialization) { + const auto scenarios = reader_scenarios(); + EXPECT_TRUE(std::ranges::any_of(scenarios, [](const ReaderScenario& scenario) { + return scenario.operation == ReaderOperation::COMPLEX_RESIDUAL_SCAN && + scenario.encoding == Encoding::PLAIN && scenario.selectivity_percent == 10 && + scenario.schema_width == 32; + })); +} + TEST(ParquetBenchmarkScenariosTest, ReaderMatrixCoversFixedWidthRawFilterAxes) { const auto scenarios = reader_scenarios(); for (const auto encoding : {Encoding::BYTE_STREAM_SPLIT, Encoding::DELTA_BINARY_PACKED}) { @@ -128,6 +199,24 @@ TEST(ParquetBenchmarkScenariosTest, ReaderMatrixCoversFixedWidthRawFilterAxes) { } } +TEST(ParquetBenchmarkScenariosTest, ReaderMatrixCoversTypedDictionaryFilterAxes) { + const auto scenarios = reader_scenarios(); + for (const auto value_type : {ValueType::INT64, ValueType::BYTE_ARRAY}) { + for (const int selectivity : {10, 50}) { + for (const auto projection : + {Projection::PREDICATE_ONLY, Projection::PREDICATE_PROJECTED}) { + EXPECT_TRUE(std::ranges::any_of(scenarios, [&](const ReaderScenario& scenario) { + return scenario.operation == ReaderOperation::PREDICATE_SCAN && + scenario.encoding == Encoding::DICTIONARY && + scenario.value_type == value_type && + scenario.selectivity_percent == selectivity && + scenario.projection == projection; + })) << "missing typed dictionary filter axis"; + } + } + } +} + TEST(ParquetBenchmarkScenariosTest, SelectionPlanDistinguishesClusteredAndSparseRuns) { const auto clustered = make_selection_plan(1000, 10, Pattern::CLUSTERED); EXPECT_EQ(clustered.total_rows, 1000); @@ -144,5 +233,18 @@ TEST(ParquetBenchmarkScenariosTest, SelectionPlanDistinguishesClusteredAndSparse EXPECT_EQ(make_selection_plan(1000, 100, Pattern::ALTERNATING).ranges.size(), 1); } +TEST(ParquetBenchmarkScenariosTest, SelectedRowVisitorPreservesRangeOrderAndBoundaries) { + const SelectionPlan plan { + .total_rows = 12, + .selected_rows = 5, + .ranges = {{.first = 0, .count = 2}, + {.first = 5, .count = 1}, + {.first = 9, .count = 2}}, + }; + std::vector rows; + visit_selected_rows(plan, [&](size_t row) { rows.push_back(row); }); + EXPECT_EQ(rows, (std::vector {0, 1, 5, 9, 10})); +} + } // namespace } // namespace doris::parquet_benchmark diff --git a/be/test/format_v2/parquet/parquet_reader_control_test.cpp b/be/test/format_v2/parquet/parquet_reader_control_test.cpp index 75e6545906f263..9de951ec5e47dd 100644 --- a/be/test/format_v2/parquet/parquet_reader_control_test.cpp +++ b/be/test/format_v2/parquet/parquet_reader_control_test.cpp @@ -177,6 +177,13 @@ TEST(SelectionVectorTest, MaterializedFilterIsReusedUntilSelectionChanges) { std::vector({0, 1, 1, 0})); } +TEST(SelectionVectorTest, IdentitySelectionDoesNotMaterializeFilter) { + SelectionVector selection(4); + const uint8_t* filter = reinterpret_cast(1); + ASSERT_TRUE(selection.materialize_filter(4, 4, &filter).ok()); + EXPECT_EQ(filter, nullptr); +} + TEST(ParquetColumnReaderControlTest, BaseSelectUsesSkipReadRanges) { CursorColumnReader reader; SelectionVector selection(3); diff --git a/be/test/format_v2/parquet/parquet_reader_test.cpp b/be/test/format_v2/parquet/parquet_reader_test.cpp index adb3b4b383c9c8..af0897d6ef2804 100644 --- a/be/test/format_v2/parquet/parquet_reader_test.cpp +++ b/be/test/format_v2/parquet/parquet_reader_test.cpp @@ -2010,6 +2010,34 @@ TEST_F(NewParquetReaderTest, ConditionCacheHitSkipsFalseGranulesBeforeColumnRead EXPECT_EQ(rows, 0); } +TEST_F(NewParquetReaderTest, ConditionCacheHitKeepsBatchAcrossAdjacentTrueGranules) { + write_condition_cache_parquet_file(_file_path); + auto reader = create_reader(); + reader->set_batch_size(ConditionCacheContext::GRANULE_SIZE * 2); + RuntimeState state {TQueryOptions(), TQueryGlobals()}; + ASSERT_TRUE(reader->init(&state).ok()); + + std::vector schema; + ASSERT_TRUE(reader->get_schema(&schema).ok()); + ASSERT_EQ(schema.size(), 1); + + auto request = std::make_shared(); + request->non_predicate_columns = {field_projection(0)}; + ASSERT_TRUE(reader->open(request).ok()); + + auto ctx = std::make_shared(); + ctx->is_hit = true; + ctx->filter_result = std::make_shared>(std::vector {true, true}); + reader->set_condition_cache_context(ctx); + + Block block = build_file_block(schema); + size_t rows = 0; + bool eof = false; + ASSERT_TRUE(reader->get_block(&block, &rows, &eof).ok()); + EXPECT_FALSE(eof); + EXPECT_EQ(rows, ConditionCacheContext::GRANULE_SIZE * 2); +} + TEST_F(NewParquetReaderTest, ReadMultipleRowGroups) { write_parquet_file(_file_path, 2); auto parquet_file_reader = ::parquet::ParquetFileReader::OpenFile(_file_path, false); diff --git a/be/test/format_v2/parquet/parquet_scan_test.cpp b/be/test/format_v2/parquet/parquet_scan_test.cpp index 9bdb0c7562f0b6..f3049a3d03adf8 100644 --- a/be/test/format_v2/parquet/parquet_scan_test.cpp +++ b/be/test/format_v2/parquet/parquet_scan_test.cpp @@ -46,11 +46,18 @@ #include "core/data_type/data_type_number.h" #include "core/data_type/data_type_string.h" #include "core/field.h" +#include "exprs/bloom_filter_func.h" +#include "exprs/create_predicate_function.h" +#include "exprs/runtime_filter_expr.h" +#include "exprs/vbloom_predicate.h" +#include "exprs/vcompound_pred.h" +#include "exprs/vdirect_in_predicate.h" #include "exprs/vectorized_fn_call.h" #include "exprs/vexpr.h" #include "exprs/vexpr_context.h" #include "exprs/vliteral.h" #include "exprs/vslot_ref.h" +#include "format_v2/expr/cast.h" #include "format_v2/expr/delete_predicate.h" #include "format_v2/file_reader.h" #include "format_v2/parquet/parquet_column_schema.h" @@ -232,6 +239,8 @@ class Int32PairSumExpr final : public VExpr { const std::string& expr_name() const override { return _expr_name; } + bool is_constant() const override { return false; } + Status execute_column_impl(VExprContext*, const Block* block, const Selector* selector, size_t count, ColumnPtr& result_column) const override { DORIS_CHECK(block != nullptr); @@ -389,7 +398,8 @@ VExprContextSPtr create_int32_zonemap_conjunct(int column_id, Int32ZoneMapExpr:: } VExprContextSPtr create_int32_function_conjunct(int column_id, const std::string& function_name, - TExprOpcode::type opcode, int32_t value) { + TExprOpcode::type opcode, int32_t value, + bool mark_prepared = true) { const auto int_type = std::make_shared(); const auto nullable_int_type = make_nullable(int_type); const auto result_type = make_nullable(std::make_shared()); @@ -415,11 +425,75 @@ VExprContextSPtr create_int32_function_conjunct(int column_id, const std::string auto context = VExprContext::create_shared(std::move(root)); // Direct evaluation does not execute the expression, but a fallback must still fail this test // instead of silently using an unprepared test-only context. - context->_prepared = true; - context->_opened = true; + if (mark_prepared) { + context->_prepared = true; + context->_opened = true; + } return context; } +VExprContextSPtr create_int64_function_conjunct(int column_id, const std::string& function_name, + TExprOpcode::type opcode, int64_t value) { + const auto bigint_type = std::make_shared(); + const auto nullable_bigint_type = make_nullable(bigint_type); + const auto result_type = make_nullable(std::make_shared()); + TFunctionName fn_name; + fn_name.__set_function_name(function_name); + TFunction fn; + fn.__set_name(fn_name); + fn.__set_binary_type(TFunctionBinaryType::BUILTIN); + fn.__set_arg_types({nullable_bigint_type->to_thrift(), bigint_type->to_thrift()}); + fn.__set_ret_type(result_type->to_thrift()); + fn.__set_has_var_args(false); + TExprNode node; + node.__set_node_type(TExprNodeType::BINARY_PRED); + node.__set_opcode(opcode); + node.__set_type(result_type->to_thrift()); + node.__set_fn(fn); + node.__set_num_children(2); + node.__set_is_nullable(true); + auto root = VectorizedFnCall::create_shared(node); + root->add_child( + VSlotRef::create_shared(column_id, column_id, -1, nullable_bigint_type, "dict_bigint")); + root->add_child(VLiteral::create_shared(bigint_type, Field::create_field(value))); + return VExprContext::create_shared(std::move(root)); +} + +VExprContextSPtr create_string_function_conjunct(int column_id, const std::string& function_name, + TExprOpcode::type opcode, const std::string& value, + bool literal_on_left = false) { + const DataTypePtr string_type = std::make_shared(); + const auto nullable_string_type = make_nullable(string_type); + const auto result_type = make_nullable(std::make_shared()); + TFunctionName fn_name; + fn_name.__set_function_name(function_name); + TFunction fn; + fn.__set_name(fn_name); + fn.__set_binary_type(TFunctionBinaryType::BUILTIN); + fn.__set_arg_types({nullable_string_type->to_thrift(), string_type->to_thrift()}); + fn.__set_ret_type(result_type->to_thrift()); + fn.__set_has_var_args(false); + TExprNode node; + node.__set_node_type(TExprNodeType::BINARY_PRED); + node.__set_opcode(opcode); + node.__set_type(result_type->to_thrift()); + node.__set_fn(fn); + node.__set_num_children(2); + node.__set_is_nullable(true); + auto root = VectorizedFnCall::create_shared(node); + auto slot = + VSlotRef::create_shared(column_id, column_id, -1, nullable_string_type, "dict_text"); + auto literal = VLiteral::create_shared(string_type, Field::create_field(value)); + if (literal_on_left) { + root->add_child(std::move(literal)); + root->add_child(std::move(slot)); + } else { + root->add_child(std::move(slot)); + root->add_child(std::move(literal)); + } + return VExprContext::create_shared(std::move(root)); +} + VExprContextSPtr create_int32_mod_greater_than_conjunct(int column_id) { const auto int_type = std::make_shared(); const auto nullable_int_type = make_nullable(int_type); @@ -470,11 +544,162 @@ VExprContextSPtr create_int32_pair_sum_conjunct(int left_column_id, int right_co std::make_shared(left_column_id, right_column_id, upper_bound)); } +VExprContextSPtr create_and_conjunct(VExprSPtr left, VExprSPtr right) { + const auto result_type = left->data_type()->is_nullable() || right->data_type()->is_nullable() + ? make_nullable(std::make_shared()) + : std::make_shared(); + TExprNode node; + node.__set_node_type(TExprNodeType::COMPOUND_PRED); + node.__set_opcode(TExprOpcode::COMPOUND_AND); + node.__set_type(result_type->to_thrift()); + node.__set_num_children(2); + node.__set_is_nullable(result_type->is_nullable()); + auto compound = VCompoundPred::create_shared(node); + compound->add_child(std::move(left)); + compound->add_child(std::move(right)); + return VExprContext::create_shared(std::move(compound)); +} + +VExprSPtr create_binary_predicate(const std::string& function_name, TExprOpcode::type opcode, + VExprSPtr left, VExprSPtr right) { + const auto result_type = left->data_type()->is_nullable() || right->data_type()->is_nullable() + ? make_nullable(std::make_shared()) + : std::make_shared(); + TFunctionName name; + name.__set_function_name(function_name); + TFunction function; + function.__set_name(name); + function.__set_binary_type(TFunctionBinaryType::BUILTIN); + function.__set_arg_types({left->data_type()->to_thrift(), right->data_type()->to_thrift()}); + function.__set_ret_type(result_type->to_thrift()); + function.__set_has_var_args(false); + TExprNode node; + node.__set_node_type(TExprNodeType::BINARY_PRED); + node.__set_opcode(opcode); + node.__set_type(result_type->to_thrift()); + node.__set_fn(function); + node.__set_num_children(2); + node.__set_is_nullable(result_type->is_nullable()); + auto predicate = VectorizedFnCall::create_shared(node); + predicate->add_child(std::move(left)); + predicate->add_child(std::move(right)); + return predicate; +} + +VExprSPtr create_int32_slot_comparison(const std::string& function_name, TExprOpcode::type opcode, + int left_column_id, int right_column_id, + const DataTypePtr& type) { + return create_binary_predicate( + function_name, opcode, + VSlotRef::create_shared(left_column_id, left_column_id, -1, type, + "c" + std::to_string(left_column_id)), + VSlotRef::create_shared(right_column_id, right_column_id, -1, type, + "c" + std::to_string(right_column_id))); +} + VExprContextSPtr create_int32_direct_greater_conjunct(int column_id, int32_t lower_bound) { return VExprContext::create_shared( std::make_shared(column_id, lower_bound)); } +TExprNode make_runtime_in_node() { + TExprNode node; + node.__set_type(std::make_shared()->to_thrift()); + node.__set_node_type(TExprNodeType::IN_PRED); + node.in_predicate.__set_is_not_in(false); + node.__set_opcode(TExprOpcode::FILTER_IN); + node.__set_is_nullable(false); + return node; +} + +VExprContextSPtr wrap_runtime_filter(VExprSPtr impl, const TExprNode& node, int filter_id) { + auto context = VExprContext::create_shared( + RuntimeFilterExpr::create_shared(node, std::move(impl), 0.0, false, filter_id)); + context->_prepared = true; + context->_opened = true; + return context; +} + +VExprContextSPtr create_int32_runtime_in_conjunct(int column_id, const std::vector& values, + int filter_id) { + std::shared_ptr filter(create_set(PrimitiveType::TYPE_INT, false)); + for (const auto value : values) { + filter->insert(&value); + } + auto node = make_runtime_in_node(); + auto impl = VDirectInPredicate::create_shared(node, std::move(filter), true); + impl->add_child(VSlotRef::create_shared(column_id, column_id, -1, + make_nullable(std::make_shared()), + "runtime_in_key")); + return wrap_runtime_filter(std::move(impl), node, filter_id); +} + +VExprContextSPtr create_int32_runtime_bloom_conjunct(int column_id, + const std::vector& values, + int filter_id) { + std::shared_ptr filter(create_bloom_filter(TYPE_INT, false)); + RuntimeFilterParams params; + params.filter_type = RuntimeFilterType::BLOOM_FILTER; + params.column_return_type = TYPE_INT; + params.bloom_filter_size = 1024; + filter->init_params(¶ms); + EXPECT_TRUE(filter->init_with_fixed_length(1024).ok()); + auto build_values = ColumnInt32::create(); + for (const auto value : values) { + build_values->insert_value(value); + } + filter->insert_fixed_len(std::move(build_values), 0); + + auto node = make_runtime_in_node(); + node.__set_node_type(TExprNodeType::BLOOM_PRED); + node.__set_opcode(TExprOpcode::RT_FILTER); + auto impl = VBloomPredicate::create_shared(node); + impl->set_filter(std::move(filter)); + impl->add_child(VSlotRef::create_shared(column_id, column_id, -1, + make_nullable(std::make_shared()), + "runtime_bloom_key")); + return wrap_runtime_filter(std::move(impl), node, filter_id); +} + +VExprContextSPtr create_string_runtime_bloom_conjunct(int column_id, + const std::vector& values, + int filter_id) { + std::shared_ptr filter(create_bloom_filter(TYPE_STRING, false)); + RuntimeFilterParams params; + params.filter_type = RuntimeFilterType::BLOOM_FILTER; + params.column_return_type = TYPE_STRING; + params.bloom_filter_size = 1024; + filter->init_params(¶ms); + EXPECT_TRUE(filter->init_with_fixed_length(1024).ok()); + auto build_values = ColumnString::create(); + for (const auto& value : values) { + build_values->insert_data(value.data(), value.size()); + } + filter->insert_fixed_len(std::move(build_values), 0); + + auto node = make_runtime_in_node(); + node.__set_node_type(TExprNodeType::BLOOM_PRED); + node.__set_opcode(TExprOpcode::RT_FILTER); + auto impl = VBloomPredicate::create_shared(node); + impl->set_filter(std::move(filter)); + impl->add_child(VSlotRef::create_shared(column_id, column_id, -1, + make_nullable(std::make_shared()), + "runtime_bloom_key")); + return wrap_runtime_filter(std::move(impl), node, filter_id); +} + +VExprContextSPtr create_int32_runtime_comparison_conjunct(int column_id, + const std::string& function_name, + TExprOpcode::type opcode, int32_t value, + int filter_id) { + auto impl_context = + create_int32_function_conjunct(column_id, function_name, opcode, value, true); + TExprNode node; + node.__set_type(std::make_shared()->to_thrift()); + node.__set_is_nullable(false); + return wrap_runtime_filter(impl_context->root(), node, filter_id); +} + VExprContextSPtr create_int64_direct_greater_conjunct(int column_id, int64_t lower_bound) { return VExprContext::create_shared( std::make_shared(column_id, lower_bound)); @@ -504,6 +729,14 @@ std::shared_ptr build_int32_array(const std::vector& valu return finish_array(&builder); } +std::shared_ptr build_int64_array(const std::vector& values) { + arrow::Int64Builder builder; + for (const auto value : values) { + EXPECT_TRUE(builder.Append(value).ok()); + } + return finish_array(&builder); +} + std::shared_ptr build_uint32_array(const std::vector& values) { arrow::UInt32Builder builder; for (const auto value : values) { @@ -635,6 +868,72 @@ void write_int_pair_parquet_file(const std::string& file_path, int64_t row_group write_table(file_path, table, row_group_size, false, false, enable_statistics, encoding); } +void write_dictionary_int_pair_parquet_file(const std::string& file_path) { + auto schema = arrow::schema({ + arrow::field("id", arrow::int32(), false), + arrow::field("score", arrow::int32(), false), + }); + auto table = arrow::Table::Make(schema, {build_int32_array({1, 2, 3, 4, 5, 6}), + build_int32_array({10, 20, 30, 40, 50, 60})}); + write_table(file_path, table, 6, true, false, false); +} + +void write_dictionary_bigint_pair_parquet_file(const std::string& file_path) { + auto schema = arrow::schema({ + arrow::field("id", arrow::int64(), false), + arrow::field("score", arrow::int32(), false), + }); + auto table = arrow::Table::Make(schema, {build_int64_array({1, 2, 3, 4, 5, 6}), + build_int32_array({10, 20, 30, 40, 50, 60})}); + write_table(file_path, table, 6, true, false, false); +} + +void write_dictionary_string_pair_parquet_file(const std::string& file_path) { + auto schema = arrow::schema({ + arrow::field("text", arrow::utf8(), false), + arrow::field("score", arrow::int32(), false), + }); + auto table = + arrow::Table::Make(schema, {build_string_array({"alpha", "bravo", "charlie", "delta"}), + build_int32_array({10, 20, 30, 40})}); + write_table(file_path, table, 4, true, false, false); +} + +void write_int_triple_parquet_file(const std::string& file_path) { + auto schema = arrow::schema({ + arrow::field("left", arrow::int32(), false), + arrow::field("middle", arrow::int32(), false), + arrow::field("right", arrow::int32(), false), + }); + auto table = arrow::Table::Make(schema, {build_int32_array({1, 2, 3, 4, 5, 6}), + build_int32_array({10, 20, 30, 0, 0, 0}), + build_int32_array({100, 200, 300, 400, 500, 600})}); + write_table(file_path, table, 6, false, false, false); +} + +void write_int_columns_parquet_file(const std::string& file_path, int column_count) { + std::vector> fields; + std::vector> columns; + fields.reserve(column_count); + columns.reserve(column_count); + for (int column = 0; column < column_count; ++column) { + fields.push_back(arrow::field("c" + std::to_string(column), arrow::int32(), false)); + columns.push_back(build_int32_array({1, 2, 3, 4, 5, 6})); + } + write_table(file_path, arrow::Table::Make(arrow::schema(std::move(fields)), std::move(columns)), + 6, false, false, false); +} + +void write_int_pair_and_string_parquet_file(const std::string& file_path) { + auto schema = arrow::schema({arrow::field("left", arrow::int32(), false), + arrow::field("right", arrow::int32(), false), + arrow::field("text", arrow::utf8(), false)}); + auto table = + arrow::Table::Make(schema, {build_int32_array({1, 2, 3}), build_int32_array({0, 3, 4}), + build_string_array({"bad", "2", "3"})}); + write_table(file_path, table, 3, false, false, false); +} + void write_uint32_pair_parquet_file(const std::string& file_path) { auto schema = arrow::schema({arrow::field("id", arrow::uint32(), false), arrow::field("score", arrow::int32(), false)}); @@ -1349,8 +1648,7 @@ TEST_F(ParquetScanTest, PredicateOnlyGlobalRowIdKeepsSignedFileLocalId) { Block block = build_file_block(schema); size_t rows = 0; bool eof = false; - const auto status = reader->get_block(&block, &rows, &eof); - ASSERT_TRUE(status.ok()) << status; + ASSERT_TRUE(reader->get_block(&block, &rows, &eof).ok()); EXPECT_EQ(rows, 6); EXPECT_EQ(int32_data_column(*block.get_by_position(0).column).get_data(), (ColumnInt32::Container {1, 2, 3, 4, 5, 6})); @@ -1372,7 +1670,8 @@ TEST_F(ParquetScanTest, EmptyScanPlanReturnsEofWithoutReadingColumns) { Block block = build_file_block(schema); size_t rows = 0; bool eof = false; - ASSERT_TRUE(reader->get_block(&block, &rows, &eof).ok()); + const auto status = reader->get_block(&block, &rows, &eof); + ASSERT_TRUE(status.ok()) << status; EXPECT_EQ(rows, 0); EXPECT_TRUE(eof); } @@ -1485,6 +1784,149 @@ TEST_F(ParquetScanTest, PredicateColumnsSkipUnreadColumnsWhenFirstPredicateFilte EXPECT_EQ(counter_value(profile, "ReaderSkipRows"), 6); } +TEST_F(ParquetScanTest, ComplexResidualSkipsColumnsAfterEarlierAndChildFiltersAll) { + write_int_triple_parquet_file(_file_path); + RuntimeProfile profile("profile"); + auto reader = create_reader(0, -1, &profile); + RuntimeState state {TQueryOptions(), TQueryGlobals()}; + ASSERT_TRUE(reader->init(&state).ok()); + + std::vector schema; + ASSERT_TRUE(reader->get_schema(&schema).ok()); + auto request = std::make_shared(); + format::FileScanRequestBuilder request_builder(request.get()); + ASSERT_TRUE(request_builder.add_predicate_column(format::LocalColumnId(0)).ok()); + ASSERT_TRUE(request_builder.add_predicate_column(format::LocalColumnId(1)).ok()); + ASSERT_TRUE(request_builder.add_predicate_column(format::LocalColumnId(2)).ok()); + auto context = create_and_conjunct( + create_int32_slot_comparison("eq", TExprOpcode::EQ, 0, 1, schema[0].type), + create_int32_slot_comparison("lt", TExprOpcode::LT, 1, 2, schema[1].type)); + ASSERT_TRUE(context->prepare(&state, RowDescriptor()).ok()); + ASSERT_TRUE(context->open(&state).ok()); + request->conjuncts.push_back(context); + ASSERT_TRUE(reader->open(request).ok()); + + Block block = build_file_block(schema); + size_t rows = 0; + bool eof = false; + ASSERT_TRUE(reader->get_block(&block, &rows, &eof).ok()); + EXPECT_EQ(rows, 0); + EXPECT_EQ(counter_value(profile, "ReaderReadRows"), 12); + EXPECT_EQ(counter_value(profile, "ReaderSelectRows"), 0); + EXPECT_EQ(counter_value(profile, "ReaderSkipRows"), 6); + context->close(); +} + +TEST_F(ParquetScanTest, ComplexResidualSelectsLaterColumnsForSurvivingRows) { + write_int_triple_parquet_file(_file_path); + RuntimeProfile profile("profile"); + auto reader = create_reader(0, -1, &profile); + RuntimeState state {TQueryOptions(), TQueryGlobals()}; + ASSERT_TRUE(reader->init(&state).ok()); + + std::vector schema; + ASSERT_TRUE(reader->get_schema(&schema).ok()); + auto request = std::make_shared(); + format::FileScanRequestBuilder request_builder(request.get()); + ASSERT_TRUE(request_builder.add_predicate_column(format::LocalColumnId(0)).ok()); + ASSERT_TRUE(request_builder.add_predicate_column(format::LocalColumnId(1)).ok()); + ASSERT_TRUE(request_builder.add_predicate_column(format::LocalColumnId(2)).ok()); + auto context = create_and_conjunct( + create_int32_slot_comparison("gt", TExprOpcode::GT, 1, 0, schema[1].type), + create_int32_slot_comparison("lt", TExprOpcode::LT, 1, 2, schema[1].type)); + ASSERT_TRUE(context->prepare(&state, RowDescriptor()).ok()); + ASSERT_TRUE(context->open(&state).ok()); + request->conjuncts.push_back(context); + ASSERT_TRUE(reader->open(request).ok()); + + Block block = build_file_block(schema); + size_t rows = 0; + bool eof = false; + ASSERT_TRUE(reader->get_block(&block, &rows, &eof).ok()); + ASSERT_EQ(rows, 3); + EXPECT_EQ(int32_data_column(*block.get_by_position(0).column).get_data(), + (ColumnInt32::Container {1, 2, 3})); + EXPECT_EQ(int32_data_column(*block.get_by_position(2).column).get_data(), + (ColumnInt32::Container {100, 200, 300})); + EXPECT_EQ(counter_value(profile, "ReaderReadRows"), 15); + EXPECT_EQ(counter_value(profile, "ReaderSelectRows"), 3); + EXPECT_EQ(counter_value(profile, "ReaderSkipRows"), 3); + context->close(); +} + +TEST_F(ParquetScanTest, AllPassResidualChainAlignsEachColumnOnce) { + write_int_columns_parquet_file(_file_path, 6); + RuntimeProfile profile("profile"); + auto reader = create_reader(0, -1, &profile); + RuntimeState state {TQueryOptions(), TQueryGlobals()}; + ASSERT_TRUE(reader->init(&state).ok()); + + std::vector schema; + ASSERT_TRUE(reader->get_schema(&schema).ok()); + auto request = std::make_shared(); + format::FileScanRequestBuilder request_builder(request.get()); + for (int column = 0; column < 6; ++column) { + ASSERT_TRUE(request_builder.add_predicate_column(format::LocalColumnId(column)).ok()); + } + for (int column = 0; column < 6; column += 2) { + auto context = VExprContext::create_shared(create_int32_slot_comparison( + "eq", TExprOpcode::EQ, column, column + 1, schema[column].type)); + ASSERT_TRUE(context->prepare(&state, RowDescriptor()).ok()); + ASSERT_TRUE(context->open(&state).ok()); + request->conjuncts.push_back(std::move(context)); + } + ASSERT_TRUE(reader->open(request).ok()); + + Block block = build_file_block(schema); + size_t rows = 0; + bool eof = false; + const auto status = reader->get_block(&block, &rows, &eof); + ASSERT_TRUE(status.ok()) << status; + ASSERT_EQ(rows, 6); + auto* alignment_columns = profile.get_counter("PredicateAlignmentColumns"); + ASSERT_NE(alignment_columns, nullptr); + EXPECT_EQ(alignment_columns->value(), 6); +} + +TEST_F(ParquetScanTest, LocalizedStrictCastPreservesRejectedRowError) { + write_int_pair_and_string_parquet_file(_file_path); + RuntimeProfile profile("profile"); + auto reader = create_reader(0, -1, &profile); + TQueryOptions query_options; + query_options.__set_enable_strict_cast(true); + RuntimeState state {query_options, TQueryGlobals()}; + ASSERT_TRUE(reader->init(&state).ok()); + + std::vector schema; + ASSERT_TRUE(reader->get_schema(&schema).ok()); + auto request = std::make_shared(); + format::FileScanRequestBuilder request_builder(request.get()); + for (int column = 0; column < 3; ++column) { + ASSERT_TRUE(request_builder.add_predicate_column(format::LocalColumnId(column)).ok()); + } + + auto cast = format::Cast::create_shared(make_nullable(std::make_shared())); + cast->add_child(VSlotRef::create_shared(2, 2, -1, schema[2].type, "text")); + auto first = create_int32_slot_comparison("lt", TExprOpcode::LT, 0, 1, schema[0].type); + auto second = + create_binary_predicate("eq", TExprOpcode::EQ, std::move(cast), + VLiteral::create_shared(std::make_shared(), + Field::create_field(2))); + auto context = create_and_conjunct(std::move(first), std::move(second)); + EXPECT_FALSE(context->root()->is_safe_to_execute_on_selected_rows()); + ASSERT_TRUE(context->prepare(&state, RowDescriptor()).ok()); + ASSERT_TRUE(context->open(&state).ok()); + request->conjuncts.push_back(context); + ASSERT_TRUE(reader->open(request).ok()); + + Block block = build_file_block(schema); + size_t rows = 0; + bool eof = false; + const auto status = reader->get_block(&block, &rows, &eof); + EXPECT_FALSE(status.ok()); + context->close(); +} + TEST_F(ParquetScanTest, PredicateOnlyColumnDropsPayloadAfterFiltering) { write_int_pair_parquet_file(_file_path, 6, false); RuntimeProfile profile("profile"); @@ -1545,6 +1987,300 @@ TEST_F(ParquetScanTest, PredicateOnlyPlainComparisonUsesPhysicalDirectPath) { EXPECT_EQ(counter_value(profile, "PredicateCompactionBytes"), 0); } +TEST_F(ParquetScanTest, PredicateOnlyPlainRuntimeFiltersUsePhysicalDirectPath) { + write_int_pair_parquet_file(_file_path, 6, false); + RuntimeProfile profile("profile"); + auto reader = create_reader(0, -1, &profile); + RuntimeState state {TQueryOptions(), TQueryGlobals()}; + ASSERT_TRUE(reader->init(&state).ok()); + + std::vector schema; + ASSERT_TRUE(reader->get_schema(&schema).ok()); + auto request = std::make_shared(); + format::FileScanRequestBuilder request_builder(request.get()); + ASSERT_TRUE(request_builder.add_predicate_column(format::LocalColumnId(0)).ok()); + ASSERT_TRUE(request_builder.add_non_predicate_column(format::LocalColumnId(1)).ok()); + request->predicate_only_columns.push_back(format::LocalColumnId(0)); + request->conjuncts.push_back( + create_int32_runtime_comparison_conjunct(0, "gt", TExprOpcode::GT, 2, 7)); + request->conjuncts.push_back(create_int32_runtime_in_conjunct(0, {3, 5, 6}, 8)); + ASSERT_TRUE(reader->open(request).ok()); + + Block block = build_file_block(schema); + size_t rows = 0; + bool eof = false; + ASSERT_TRUE(reader->get_block(&block, &rows, &eof).ok()); + ASSERT_EQ(rows, 3); + EXPECT_EQ(int32_data_column(*block.get_by_position(1).column).get_data(), + (ColumnInt32::Container {30, 50, 60})); + EXPECT_EQ(counter_value(profile, "FixedWidthPredicateDirectBatches"), 1); + EXPECT_EQ(counter_value(profile, "FixedWidthPredicateDirectRows"), 6); +} + +TEST_F(ParquetScanTest, PredicateOnlyPlainBloomRuntimeFilterUsesPhysicalDirectPath) { + write_int_pair_parquet_file(_file_path, 6, false); + RuntimeProfile profile("profile"); + auto reader = create_reader(0, -1, &profile); + RuntimeState state {TQueryOptions(), TQueryGlobals()}; + ASSERT_TRUE(reader->init(&state).ok()); + + std::vector schema; + ASSERT_TRUE(reader->get_schema(&schema).ok()); + auto request = std::make_shared(); + format::FileScanRequestBuilder request_builder(request.get()); + ASSERT_TRUE(request_builder.add_predicate_column(format::LocalColumnId(0)).ok()); + ASSERT_TRUE(request_builder.add_non_predicate_column(format::LocalColumnId(1)).ok()); + request->predicate_only_columns.push_back(format::LocalColumnId(0)); + request->conjuncts.push_back(create_int32_runtime_bloom_conjunct(0, {3, 5, 6}, 9)); + ASSERT_TRUE(reader->open(request).ok()); + + Block block = build_file_block(schema); + size_t rows = 0; + bool eof = false; + ASSERT_TRUE(reader->get_block(&block, &rows, &eof).ok()); + ASSERT_EQ(rows, 3); + EXPECT_EQ(int32_data_column(*block.get_by_position(1).column).get_data(), + (ColumnInt32::Container {30, 50, 60})); + EXPECT_EQ(counter_value(profile, "FixedWidthPredicateDirectBatches"), 1); + EXPECT_EQ(counter_value(profile, "FixedWidthPredicateDirectRows"), 6); +} + +TEST_F(ParquetScanTest, PredicateOnlyDictionaryRangeSkipsTypedValueMaterialization) { + write_dictionary_int_pair_parquet_file(_file_path); + RuntimeProfile profile("profile"); + auto reader = create_reader(0, -1, &profile); + RuntimeState state {TQueryOptions(), TQueryGlobals()}; + ASSERT_TRUE(reader->init(&state).ok()); + + std::vector schema; + ASSERT_TRUE(reader->get_schema(&schema).ok()); + auto request = std::make_shared(); + format::FileScanRequestBuilder request_builder(request.get()); + ASSERT_TRUE(request_builder.add_predicate_column(format::LocalColumnId(0)).ok()); + ASSERT_TRUE(request_builder.add_non_predicate_column(format::LocalColumnId(1)).ok()); + request->predicate_only_columns.push_back(format::LocalColumnId(0)); + auto conjunct = create_int32_function_conjunct(0, "gt", TExprOpcode::GT, 2, false); + ASSERT_TRUE(conjunct->prepare(&state, RowDescriptor()).ok()); + ASSERT_TRUE(conjunct->open(&state).ok()); + request->conjuncts.push_back(conjunct); + ASSERT_TRUE(reader->open(request).ok()); + + Block block = build_file_block(schema); + size_t rows = 0; + bool eof = false; + ASSERT_TRUE(reader->get_block(&block, &rows, &eof).ok()); + ASSERT_EQ(rows, 4); + EXPECT_EQ(int32_data_column(*block.get_by_position(1).column).get_data(), + (ColumnInt32::Container {30, 40, 50, 60})); + EXPECT_EQ(block.get_by_position(0).column->size(), rows); + EXPECT_EQ(counter_value(profile, "DictFilterCandidateColumns"), 1); + EXPECT_EQ(counter_value(profile, "DictFilterColumns"), 1); + EXPECT_EQ(counter_value(profile, "RowsFilteredByDictFilter"), 2); + EXPECT_EQ(counter_value(profile, "DictionaryPredicateDirectBatches"), 1); + EXPECT_EQ(counter_value(profile, "DictionaryPredicateDirectRows"), 6); + EXPECT_EQ(counter_value(profile, "DictionaryPredicateProjectedRows"), 0); + EXPECT_EQ(counter_value(profile, "PredicateCompactionCount"), 0); + conjunct->close(); +} + +TEST_F(ParquetScanTest, PredicateOnlyDictionaryBloomRuntimeFilterUsesTypedValues) { + write_dictionary_int_pair_parquet_file(_file_path); + RuntimeProfile profile("profile"); + auto reader = create_reader(0, -1, &profile); + RuntimeState state {TQueryOptions(), TQueryGlobals()}; + ASSERT_TRUE(reader->init(&state).ok()); + + std::vector schema; + ASSERT_TRUE(reader->get_schema(&schema).ok()); + auto request = std::make_shared(); + format::FileScanRequestBuilder request_builder(request.get()); + ASSERT_TRUE(request_builder.add_predicate_column(format::LocalColumnId(0)).ok()); + ASSERT_TRUE(request_builder.add_non_predicate_column(format::LocalColumnId(1)).ok()); + request->predicate_only_columns.push_back(format::LocalColumnId(0)); + request->conjuncts.push_back(create_int32_runtime_bloom_conjunct(0, {3, 5, 6}, 10)); + ASSERT_TRUE(reader->open(request).ok()); + + Block block = build_file_block(schema); + size_t rows = 0; + bool eof = false; + ASSERT_TRUE(reader->get_block(&block, &rows, &eof).ok()); + ASSERT_EQ(rows, 3); + EXPECT_EQ(int32_data_column(*block.get_by_position(1).column).get_data(), + (ColumnInt32::Container {30, 50, 60})); + EXPECT_EQ(counter_value(profile, "DictFilterColumns"), 1); + EXPECT_EQ(counter_value(profile, "DictFilterTypedCompareColumns"), 1); + EXPECT_EQ(counter_value(profile, "DictionaryPredicateDirectRows"), 6); +} + +TEST_F(ParquetScanTest, PredicateOnlyStringDictionaryBloomRuntimeFilterUsesDictionaryValues) { + write_dictionary_string_pair_parquet_file(_file_path); + RuntimeProfile profile("profile"); + auto reader = create_reader(0, -1, &profile); + RuntimeState state {TQueryOptions(), TQueryGlobals()}; + ASSERT_TRUE(reader->init(&state).ok()); + + std::vector schema; + ASSERT_TRUE(reader->get_schema(&schema).ok()); + auto request = std::make_shared(); + format::FileScanRequestBuilder request_builder(request.get()); + ASSERT_TRUE(request_builder.add_predicate_column(format::LocalColumnId(0)).ok()); + ASSERT_TRUE(request_builder.add_non_predicate_column(format::LocalColumnId(1)).ok()); + request->predicate_only_columns.push_back(format::LocalColumnId(0)); + request->conjuncts.push_back(create_string_runtime_bloom_conjunct(0, {"bravo", "delta"}, 11)); + ASSERT_TRUE(reader->open(request).ok()); + + Block block = build_file_block(schema); + size_t rows = 0; + bool eof = false; + ASSERT_TRUE(reader->get_block(&block, &rows, &eof).ok()); + ASSERT_EQ(rows, 2); + EXPECT_EQ(int32_data_column(*block.get_by_position(1).column).get_data(), + (ColumnInt32::Container {20, 40})); + EXPECT_EQ(counter_value(profile, "DictFilterColumns"), 1); + EXPECT_EQ(counter_value(profile, "DictionaryPredicateDirectRows"), 4); +} + +TEST_F(ParquetScanTest, ProjectedDictionaryRangeGathersOnlySurvivors) { + write_dictionary_int_pair_parquet_file(_file_path); + RuntimeProfile profile("profile"); + auto reader = create_reader(0, -1, &profile); + RuntimeState state {TQueryOptions(), TQueryGlobals()}; + ASSERT_TRUE(reader->init(&state).ok()); + + std::vector schema; + ASSERT_TRUE(reader->get_schema(&schema).ok()); + auto request = std::make_shared(); + format::FileScanRequestBuilder request_builder(request.get()); + ASSERT_TRUE(request_builder.add_predicate_column(format::LocalColumnId(0)).ok()); + ASSERT_TRUE(request_builder.add_non_predicate_column(format::LocalColumnId(1)).ok()); + auto conjunct = create_int32_function_conjunct(0, "gt", TExprOpcode::GT, 2, false); + ASSERT_TRUE(conjunct->prepare(&state, RowDescriptor()).ok()); + ASSERT_TRUE(conjunct->open(&state).ok()); + request->conjuncts.push_back(conjunct); + ASSERT_TRUE(reader->open(request).ok()); + + Block block = build_file_block(schema); + size_t rows = 0; + bool eof = false; + ASSERT_TRUE(reader->get_block(&block, &rows, &eof).ok()); + ASSERT_EQ(rows, 4); + EXPECT_EQ(int32_data_column(*block.get_by_position(0).column).get_data(), + (ColumnInt32::Container {3, 4, 5, 6})); + EXPECT_EQ(int32_data_column(*block.get_by_position(1).column).get_data(), + (ColumnInt32::Container {30, 40, 50, 60})); + EXPECT_EQ(counter_value(profile, "DictFilterColumns"), 1); + EXPECT_EQ(counter_value(profile, "DictionaryPredicateDirectBatches"), 1); + EXPECT_EQ(counter_value(profile, "DictionaryPredicateDirectRows"), 6); + EXPECT_EQ(counter_value(profile, "DictionaryPredicateProjectedRows"), 4); + EXPECT_EQ(counter_value(profile, "PredicateCompactionCount"), 0); + conjunct->close(); +} + +TEST_F(ParquetScanTest, ProjectedBigIntDictionaryRangeUsesFusedGather) { + write_dictionary_bigint_pair_parquet_file(_file_path); + RuntimeProfile profile("profile"); + auto reader = create_reader(0, -1, &profile); + RuntimeState state {TQueryOptions(), TQueryGlobals()}; + ASSERT_TRUE(reader->init(&state).ok()); + + std::vector schema; + ASSERT_TRUE(reader->get_schema(&schema).ok()); + auto request = std::make_shared(); + format::FileScanRequestBuilder request_builder(request.get()); + ASSERT_TRUE(request_builder.add_predicate_column(format::LocalColumnId(0)).ok()); + ASSERT_TRUE(request_builder.add_non_predicate_column(format::LocalColumnId(1)).ok()); + auto conjunct = create_int64_function_conjunct(0, "gt", TExprOpcode::GT, 2); + ASSERT_TRUE(conjunct->prepare(&state, RowDescriptor()).ok()); + ASSERT_TRUE(conjunct->open(&state).ok()); + request->conjuncts.push_back(conjunct); + ASSERT_TRUE(reader->open(request).ok()); + + Block block = build_file_block(schema); + size_t rows = 0; + bool eof = false; + ASSERT_TRUE(reader->get_block(&block, &rows, &eof).ok()); + ASSERT_EQ(rows, 4); + EXPECT_EQ(int64_data_column(*block.get_by_position(0).column).get_data(), + (ColumnInt64::Container {3, 4, 5, 6})); + EXPECT_EQ(int32_data_column(*block.get_by_position(1).column).get_data(), + (ColumnInt32::Container {30, 40, 50, 60})); + auto* fused_rows = profile.get_counter("DictionaryPredicateFusedProjectedRows"); + ASSERT_NE(fused_rows, nullptr); + EXPECT_EQ(fused_rows->value(), 4); + auto* typed_filter_columns = profile.get_counter("DictFilterTypedCompareColumns"); + ASSERT_NE(typed_filter_columns, nullptr); + EXPECT_EQ(typed_filter_columns->value(), 1); + conjunct->close(); +} + +TEST_F(ParquetScanTest, ProjectedStringDictionaryRangeGathersOnlySurvivors) { + write_dictionary_string_pair_parquet_file(_file_path); + RuntimeProfile profile("profile"); + auto reader = create_reader(0, -1, &profile); + RuntimeState state {TQueryOptions(), TQueryGlobals()}; + ASSERT_TRUE(reader->init(&state).ok()); + + std::vector schema; + ASSERT_TRUE(reader->get_schema(&schema).ok()); + auto request = std::make_shared(); + format::FileScanRequestBuilder request_builder(request.get()); + ASSERT_TRUE(request_builder.add_predicate_column(format::LocalColumnId(0)).ok()); + ASSERT_TRUE(request_builder.add_non_predicate_column(format::LocalColumnId(1)).ok()); + auto conjunct = create_string_function_conjunct(0, "gt", TExprOpcode::GT, "bravo"); + ASSERT_TRUE(conjunct->prepare(&state, RowDescriptor()).ok()); + ASSERT_TRUE(conjunct->open(&state).ok()); + request->conjuncts.push_back(conjunct); + ASSERT_TRUE(reader->open(request).ok()); + + Block block = build_file_block(schema); + size_t rows = 0; + bool eof = false; + ASSERT_TRUE(reader->get_block(&block, &rows, &eof).ok()); + ASSERT_EQ(rows, 2); + const auto& text = string_data_column(*block.get_by_position(0).column); + EXPECT_EQ(text.get_data_at(0).to_string(), "charlie"); + EXPECT_EQ(text.get_data_at(1).to_string(), "delta"); + EXPECT_EQ(int32_data_column(*block.get_by_position(1).column).get_data(), + (ColumnInt32::Container {30, 40})); + EXPECT_EQ(counter_value(profile, "DictionaryPredicateProjectedRows"), 2); + EXPECT_EQ(counter_value(profile, "PredicateCompactionCount"), 0); + auto* string_filter_columns = profile.get_counter("DictFilterStringCompareColumns"); + ASSERT_NE(string_filter_columns, nullptr); + EXPECT_EQ(string_filter_columns->value(), 1); + conjunct->close(); +} + +TEST_F(ParquetScanTest, StringDictionaryRangeNormalizesLiteralOnLeft) { + write_dictionary_string_pair_parquet_file(_file_path); + RuntimeProfile profile("profile"); + auto reader = create_reader(0, -1, &profile); + RuntimeState state {TQueryOptions(), TQueryGlobals()}; + ASSERT_TRUE(reader->init(&state).ok()); + + std::vector schema; + ASSERT_TRUE(reader->get_schema(&schema).ok()); + auto request = std::make_shared(); + format::FileScanRequestBuilder request_builder(request.get()); + ASSERT_TRUE(request_builder.add_predicate_column(format::LocalColumnId(0)).ok()); + ASSERT_TRUE(request_builder.add_non_predicate_column(format::LocalColumnId(1)).ok()); + auto conjunct = create_string_function_conjunct(0, "lt", TExprOpcode::LT, "bravo", true); + ASSERT_TRUE(conjunct->prepare(&state, RowDescriptor()).ok()); + ASSERT_TRUE(conjunct->open(&state).ok()); + request->conjuncts.push_back(conjunct); + ASSERT_TRUE(reader->open(request).ok()); + + Block block = build_file_block(schema); + size_t rows = 0; + bool eof = false; + ASSERT_TRUE(reader->get_block(&block, &rows, &eof).ok()); + ASSERT_EQ(rows, 2); + const auto& text = string_data_column(*block.get_by_position(0).column); + EXPECT_EQ(text.get_data_at(0).to_string_view(), "charlie"); + EXPECT_EQ(text.get_data_at(1).to_string_view(), "delta"); + EXPECT_EQ(counter_value(profile, "DictFilterStringCompareColumns"), 1); + conjunct->close(); +} + TEST_F(ParquetScanTest, ProjectedPlainComparisonUsesPhysicalFilterAndProjectPath) { write_int_pair_parquet_file(_file_path, 6, false); RuntimeProfile profile("profile"); diff --git a/be/test/format_v2/parquet/parquet_simd_kernels_test.cpp b/be/test/format_v2/parquet/parquet_simd_kernels_test.cpp new file mode 100644 index 00000000000000..c1a674f7aaa9f0 --- /dev/null +++ b/be/test/format_v2/parquet/parquet_simd_kernels_test.cpp @@ -0,0 +1,222 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +#include + +#include +#include +#include +#include +#include +#include +#include +#include + +#include "util/byte_stream_split.h" +#include "util/simd/parquet_kernels.h" + +namespace doris::simd { +namespace { + +TEST(ParquetSimdKernelsTest, ByteStreamSplitRestoresFourAndEightByteValues) { + for (const size_t width : {4, 8}) { + constexpr size_t rows = 67; + std::vector plain(rows * width); + for (size_t byte = 0; byte < plain.size(); ++byte) { + plain[byte] = static_cast((byte * 29 + 17) & 0xff); + } + std::vector encoded(plain.size()); + for (size_t row = 0; row < rows; ++row) { + for (size_t byte = 0; byte < width; ++byte) { + encoded[byte * rows + row] = plain[row * width + byte]; + } + } + + for (const auto [offset, count] : + {std::pair {0, rows}, {3, 31}, {17, 33}}) { + std::vector decoded(count * width); + if (!try_byte_stream_split_decode(encoded.data(), width, offset, count, rows, + decoded.data())) { + doris::byte_stream_split_decode(encoded.data(), static_cast(width), offset, + count, rows, decoded.data()); + } + EXPECT_EQ(0, memcmp(decoded.data(), plain.data() + offset * width, decoded.size())); + } + } +} + +TEST(ParquetSimdKernelsTest, ByteStreamSplitDeclinesToReplaceOptimizedScalarFallback) { + constexpr size_t rows = 191; + constexpr size_t offset = 17; + constexpr size_t count = 31; + for (const size_t width : {1, 2, 4, 8, 12, 16, 23}) { + std::vector plain(rows * width); + for (size_t byte = 0; byte < plain.size(); ++byte) { + plain[byte] = static_cast((byte * 29 + 17) & 0xff); + } + std::vector encoded(plain.size()); + for (size_t row = 0; row < rows; ++row) { + for (size_t byte = 0; byte < width; ++byte) { + encoded[byte * rows + row] = plain[row * width + byte]; + } + } + + std::vector decoded(count * width); + EXPECT_FALSE(try_byte_stream_split_decode(encoded.data(), width, offset, count, rows, + decoded.data())) + << "width=" << width; + doris::byte_stream_split_decode(encoded.data(), static_cast(width), offset, count, + rows, decoded.data()); + EXPECT_EQ(0, memcmp(decoded.data(), plain.data() + offset * width, decoded.size())) + << "width=" << width; + } +} + +template +void test_delta_prefix_sum() { + std::vector deltas(71); + for (size_t row = 0; row < deltas.size(); ++row) { + deltas[row] = static_cast((row * 7) % 19); + } + auto expected = deltas; + T expected_last = std::numeric_limits::max() - 37; + constexpr T min_delta = static_cast(-11); + using Unsigned = std::make_unsigned_t; + for (auto& value : expected) { + value = static_cast(static_cast(value) + static_cast(min_delta) + + static_cast(expected_last)); + expected_last = value; + } + + T last = std::numeric_limits::max() - 37; + delta_decode(deltas.data(), deltas.size(), min_delta, &last); + EXPECT_EQ(deltas, expected); + EXPECT_EQ(last, expected_last); +} + +TEST(ParquetSimdKernelsTest, DeltaPrefixSumPreservesParquetUnsignedOverflow) { + test_delta_prefix_sum(); + test_delta_prefix_sum(); +} + +TEST(ParquetSimdKernelsTest, DictionaryGatherHandlesTailsAndRepeatedIds) { + for (const size_t width : {4, 8}) { + constexpr size_t entries = 257; + std::vector dictionary(entries * width); + for (size_t byte = 0; byte < dictionary.size(); ++byte) { + dictionary[byte] = static_cast((byte * 13 + 5) & 0xff); + } + std::vector ids(69); + for (size_t row = 0; row < ids.size(); ++row) { + ids[row] = row % 9 == 0 ? 3 : static_cast((row * 31) % entries); + } + std::vector actual(ids.size() * width); + dictionary_gather(dictionary.data(), ids.data(), ids.size(), width, actual.data()); + for (size_t row = 0; row < ids.size(); ++row) { + EXPECT_EQ(0, memcmp(actual.data() + row * width, dictionary.data() + ids[row] * width, + width)); + } + } +} + +TEST(ParquetSimdKernelsTest, NullableExpansionIsSafeForOverlappingStorage) { + for (const size_t width : {4, 8}) { + const std::vector> null_masks { + {1, 0, 1, 0, 0, 1, 0, 1, 0, 0, 0, 1, 1, 0, 0, 1, 0}, + std::vector(17, 0), + std::vector(17, 1), + }; + for (const auto& nulls : null_masks) { + const size_t compact_count = + static_cast(std::count(nulls.begin(), nulls.end(), uint8_t {0})); + std::vector values(nulls.size() * width, 0xcd); + for (size_t row = 0; row < compact_count; ++row) { + for (size_t byte = 0; byte < width; ++byte) { + values[row * width + byte] = + static_cast((row * width + byte + 1) & 0xff); + } + } + const auto compact = values; + + expand_nullable_values(values.data(), compact_count, nulls.data(), nulls.size(), width); + size_t source = 0; + for (size_t row = 0; row < nulls.size(); ++row) { + if (nulls[row] != 0) { + for (size_t byte = 0; byte < width; ++byte) { + EXPECT_EQ(values[row * width + byte], 0); + } + } else { + for (size_t byte = 0; byte < width; ++byte) { + EXPECT_EQ(values[row * width + byte], compact[source * width + byte]) + << "row=" << row << ", byte=" << byte << ", width=" << width; + } + ++source; + } + } + } + } +} + +template +void expect_raw_comparison(const std::vector& values, T literal, RawComparisonOp op, + const std::vector& expected) { + std::vector matches(values.size(), 1); + if (matches.size() > 2) { + matches[1] = 0; + } + auto masked_expected = expected; + if (masked_expected.size() > 2) { + masked_expected[1] = 0; + } + raw_compare(reinterpret_cast(values.data()), values.size(), literal, op, + matches.data()); + EXPECT_EQ(matches, masked_expected); +} + +TEST(ParquetSimdKernelsTest, RawPredicatesPreserveExistingMaskAndDorisNanOrdering) { + const float nan = std::numeric_limits::quiet_NaN(); + const std::vector floats {-3.0F, -2.0F, -1.0F, 0.0F, 1.0F, 2.0F, + 3.0F, nan, nan, 4.0F, 5.0F}; + expect_raw_comparison(floats, 0.0F, RawComparisonOp::GT, {0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1}); + expect_raw_comparison(floats, nan, RawComparisonOp::EQ, {0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0}); + expect_raw_comparison(floats, nan, RawComparisonOp::LE, {1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1}); + + const std::vector ints {-3, -1, 0, 1, 3, 7, 9, 12, 15, 21, 27}; + expect_raw_comparison(ints, 3, RawComparisonOp::GE, {0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1}); + expect_raw_comparison(ints, 3, RawComparisonOp::NE, {1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1}); + + const double double_nan = std::numeric_limits::quiet_NaN(); + const std::vector doubles { + -std::numeric_limits::infinity(), -2.0, -1.0, 0.0, 1.0, + std::numeric_limits::infinity(), double_nan, double_nan}; + expect_raw_comparison(doubles, 0.0, RawComparisonOp::EQ, {0, 0, 0, 1, 0, 0, 0, 0}); + expect_raw_comparison(doubles, 0.0, RawComparisonOp::NE, {1, 1, 1, 0, 1, 1, 1, 1}); + expect_raw_comparison(doubles, 0.0, RawComparisonOp::LT, {1, 1, 1, 0, 0, 0, 0, 0}); + expect_raw_comparison(doubles, 0.0, RawComparisonOp::LE, {1, 1, 1, 1, 0, 0, 0, 0}); + expect_raw_comparison(doubles, 0.0, RawComparisonOp::GT, {0, 0, 0, 0, 1, 1, 1, 1}); + expect_raw_comparison(doubles, 0.0, RawComparisonOp::GE, {0, 0, 0, 1, 1, 1, 1, 1}); + expect_raw_comparison(doubles, double_nan, RawComparisonOp::EQ, {0, 0, 0, 0, 0, 0, 1, 1}); + expect_raw_comparison(doubles, double_nan, RawComparisonOp::LT, {1, 1, 1, 1, 1, 1, 0, 0}); + expect_raw_comparison(doubles, double_nan, RawComparisonOp::LE, {1, 1, 1, 1, 1, 1, 1, 1}); + + const std::vector bigints {-9, -3, -1, 0, 1, 3, 7, 11, 19}; + expect_raw_comparison(bigints, int64_t {3}, RawComparisonOp::LT, {1, 1, 1, 1, 1, 0, 0, 0, 0}); + expect_raw_comparison(bigints, int64_t {3}, RawComparisonOp::GE, {0, 0, 0, 0, 0, 1, 1, 1, 1}); +} + +} // namespace +} // namespace doris::simd diff --git a/be/test/format_v2/table/hudi_reader_test.cpp b/be/test/format_v2/table/hudi_reader_test.cpp index f5bd70292e51a3..96126281744f5a 100644 --- a/be/test/format_v2/table/hudi_reader_test.cpp +++ b/be/test/format_v2/table/hudi_reader_test.cpp @@ -37,9 +37,6 @@ #include "core/data_type/data_type_string.h" #include "core/data_type/data_type_struct.h" #include "core/field.h" -#include "exec/scan/file_scanner_v2.h" -#include "exprs/vexpr_context.h" -#include "exprs/vliteral.h" #include "format_v2/column_data.h" #include "gen_cpp/ExternalTableSchema_types.h" #include "gen_cpp/PlanNodes_types.h" @@ -144,54 +141,6 @@ class SlowInitTableReader final : public TableReader { } }; -class AppendTrackingTableReader final : public TableReader { -public: - Status append_conjuncts(const VExprContextSPtrs& conjuncts) override { - appended_conjuncts += conjuncts.size(); - owned_conjuncts += _appended_table_reader_owned_conjunct_count.value_or(conjuncts.size()); - return Status::OK(); - } - - size_t appended_conjuncts = 0; - size_t owned_conjuncts = 0; -}; - -class OneRowTableReader final : public TableReader { -public: - Status prepare_split(const SplitReadOptions&) override { return Status::OK(); } - - Status get_block(Block* block, bool* eos) override { - auto column = ColumnInt32::create(); - column->insert_value(1); - block->replace_by_position(0, std::move(column)); - *eos = false; - return Status::OK(); - } -}; - -class StatefulHybridPredicate final : public VExpr { -public: - explicit StatefulHybridPredicate(std::vector* observed_invocations) - : VExpr(std::make_shared(), false), - _observed_invocations(observed_invocations) {} - - Status execute_column_impl(VExprContext*, const Block*, const Selector*, size_t count, - ColumnPtr& result_column) const override { - _observed_invocations->push_back(_invocation++); - result_column = ColumnUInt8::create(count, 1); - return Status::OK(); - } - - const std::string& expr_name() const override { return _expr_name; } - bool is_constant() const override { return false; } - bool is_deterministic() const override { return false; } - -private: - std::vector* const _observed_invocations; - mutable int _invocation = 0; - const std::string _expr_name = "StatefulHybridPredicate"; -}; - // Scenario: FileScannerV2 Hudi native reader uses the split schema id to annotate the physical // file schema before TableColumnMapper runs. This keeps schema-evolved Hudi files on field-id // mapping, including renamed nested children. @@ -313,102 +262,6 @@ TEST(HudiHybridReaderTest, AdaptiveBatchSizeReachesBothChildReaders) { EXPECT_EQ(child_batch_sizes.second, 123); } -TEST(HudiHybridReaderTest, ReportsActiveChildMaterializedBlockStats) { - hudi::HudiHybridReader reader; - reader.TEST_install_batch_size_children(); - reader._current_split_reader = reader._native_reader.get(); - reader._native_reader->_last_materialized_block_stats = { - .has_materialized_input = true, .rows = 7, .bytes = 70, .allocated_bytes = 96}; - - const auto& stats = reader.last_materialized_block_stats(); - EXPECT_TRUE(stats.has_materialized_input); - EXPECT_EQ(stats.rows, 7); - EXPECT_EQ(stats.bytes, 70); - EXPECT_EQ(stats.allocated_bytes, 96); -} - -TEST(HudiHybridReaderTest, LateConjunctReachesInitializedNativeAndJniChildren) { - RuntimeState state {TQueryOptions(), TQueryGlobals()}; - hudi::HudiHybridReader reader; - ASSERT_TRUE(reader.init({ - .projected_columns = {}, - .conjuncts = {}, - .format = FileFormat::PARQUET, - .scan_params = nullptr, - .io_ctx = nullptr, - .runtime_state = &state, - .scanner_profile = nullptr, - }) - .ok()); - auto native_reader = std::make_unique(); - auto jni_reader = std::make_unique(); - auto* native_reader_ptr = native_reader.get(); - auto* jni_reader_ptr = jni_reader.get(); - reader._native_reader = std::move(native_reader); - reader._jni_reader = std::move(jni_reader); - - auto literal = VLiteral::create_shared(std::make_shared(), - Field::create_field(1)); - ASSERT_TRUE(reader.append_conjuncts_with_ownership( - {VExprContext::create_shared(std::move(literal))}, 0) - .ok()); - EXPECT_EQ(native_reader_ptr->appended_conjuncts, 1); - EXPECT_EQ(jni_reader_ptr->appended_conjuncts, 1); - EXPECT_EQ(native_reader_ptr->owned_conjuncts, 0); - EXPECT_EQ(jni_reader_ptr->owned_conjuncts, 0); -} - -TEST(HudiHybridReaderTest, ScannerStatefulResidualSurvivesNativeJniNativeSwitch) { - RuntimeState state {TQueryOptions(), TQueryGlobals()}; - RuntimeProfile profile("hudi_scanner_stateful_residual"); - TFileScanRangeParams scan_params; - scan_params.__set_format_type(TFileFormatType::FORMAT_PARQUET); - auto hybrid_reader = std::make_unique(); - auto* hybrid_reader_ptr = hybrid_reader.get(); - hybrid_reader_ptr->TEST_set_child_reader_factories( - [] { return std::make_unique(); }, - [] { return std::make_unique(); }); - - std::vector observed_invocations; - auto conjunct = VExprContext::create_shared( - std::make_shared(&observed_invocations)); - ASSERT_TRUE(conjunct->prepare(&state, RowDescriptor {}).ok()); - ASSERT_TRUE(conjunct->open(&state).ok()); - FileScannerV2 scanner(&state, &profile, std::move(hybrid_reader)); - scanner.TEST_set_scanner_conjuncts({std::move(conjunct)}); - - const std::vector projected_columns { - make_table_column(0, "id", std::make_shared()), - }; - ASSERT_TRUE(hybrid_reader_ptr - ->init({ - .projected_columns = projected_columns, - .conjuncts = {}, - .format = FileFormat::PARQUET, - .scan_params = &scan_params, - .io_ctx = nullptr, - .runtime_state = &state, - .scanner_profile = &profile, - }) - .ok()); - - auto run_split = [&](FileFormat format, TFileFormatType::type thrift_format) { - SplitReadOptions split; - split.current_split_format = format; - split.current_range.__set_format_type(thrift_format); - ASSERT_TRUE(hybrid_reader_ptr->prepare_split(split).ok()); - Block block = build_table_block(projected_columns); - bool eos = false; - ASSERT_TRUE(hybrid_reader_ptr->get_block(&block, &eos).ok()); - ASSERT_TRUE(scanner.TEST_filter_output_block(&block).ok()); - }; - run_split(FileFormat::PARQUET, TFileFormatType::FORMAT_PARQUET); - run_split(FileFormat::JNI, TFileFormatType::FORMAT_JNI); - run_split(FileFormat::PARQUET, TFileFormatType::FORMAT_PARQUET); - - EXPECT_EQ(observed_invocations, std::vector({0, 1, 2})); -} - TEST(HudiHybridReaderTest, AggregatesConditionCacheHitsFromBothChildren) { hudi::HudiHybridReader reader; reader.TEST_install_batch_size_children(); diff --git a/be/test/format_v2/table/iceberg_reader_test.cpp b/be/test/format_v2/table/iceberg_reader_test.cpp index c42d68933f2014..782463e335a743 100644 --- a/be/test/format_v2/table/iceberg_reader_test.cpp +++ b/be/test/format_v2/table/iceberg_reader_test.cpp @@ -1146,6 +1146,11 @@ VExprContextSPtr prepared_conjunct(RuntimeState* state, const VExprSPtr& expr) { return ctx; } +void apply_final_conjuncts(Block* block, const VExprContextSPtrs& conjuncts) { + const auto status = VExprContext::filter_block(conjuncts, block, block->columns()); + ASSERT_TRUE(status.ok()) << status; +} + TEST(IcebergV2ReaderTest, IcebergVirtualColumnsUseRowLineageMetadata) { const auto test_dir = std::filesystem::temp_directory_path() / "doris_iceberg_virtual_columns_test"; @@ -1384,6 +1389,9 @@ TEST(IcebergV2ReaderTest, IcebergRowIdPredicateFiltersAfterRowLineageMaterializa bool eos = false; ASSERT_TRUE(reader.get_block(&block, &eos).ok()); ASSERT_FALSE(eos); + ASSERT_EQ(block.rows(), 3); + + apply_final_conjuncts(&block, conjuncts); ASSERT_EQ(block.rows(), 1); expect_nullable_int64_column_values(*block.get_by_position(0).column, {1001}); expect_nullable_int64_column_values(*block.get_by_position(1).column, {77}); @@ -1435,6 +1443,9 @@ TEST(IcebergV2ReaderTest, IcebergLastUpdatedSequencePredicateFiltersAfterMateria bool eos = false; ASSERT_TRUE(reader.get_block(&block, &eos).ok()); ASSERT_FALSE(eos); + ASSERT_EQ(block.rows(), 3); + + apply_final_conjuncts(&block, conjuncts); ASSERT_EQ(block.rows(), 1); expect_nullable_int64_column_values(*block.get_by_position(0).column, {1001}); expect_nullable_int64_column_values(*block.get_by_position(1).column, {77}); diff --git a/be/test/format_v2/table/paimon_reader_test.cpp b/be/test/format_v2/table/paimon_reader_test.cpp index 77466439c6be78..06301815b49228 100644 --- a/be/test/format_v2/table/paimon_reader_test.cpp +++ b/be/test/format_v2/table/paimon_reader_test.cpp @@ -45,9 +45,6 @@ #include "core/data_type/data_type_string.h" #include "core/field.h" #include "exec/common/endian.h" -#include "exec/scan/file_scanner_v2.h" -#include "exprs/vexpr_context.h" -#include "exprs/vliteral.h" #include "format/format_common.h" #include "format/table/deletion_vector_reader.h" #include "format/table/paimon_reader.h" @@ -82,52 +79,14 @@ class SlowInitTableReader final : public TableReader { } }; -class AppendTrackingTableReader final : public TableReader { +class SplitFormatTrackingTableReader final : public TableReader { public: - Status append_conjuncts(const VExprContextSPtrs& conjuncts) override { - appended_conjuncts += conjuncts.size(); - owned_conjuncts += _appended_table_reader_owned_conjunct_count.value_or(conjuncts.size()); + Status prepare_split(const SplitReadOptions& options) override { + prepared_format = options.current_split_format; return Status::OK(); } - size_t appended_conjuncts = 0; - size_t owned_conjuncts = 0; -}; - -class OneRowTableReader final : public TableReader { -public: - Status prepare_split(const SplitReadOptions&) override { return Status::OK(); } - - Status get_block(Block* block, bool* eos) override { - auto column = ColumnInt32::create(); - column->insert_value(1); - block->replace_by_position(0, std::move(column)); - *eos = false; - return Status::OK(); - } -}; - -class StatefulHybridPredicate final : public VExpr { -public: - explicit StatefulHybridPredicate(std::vector* observed_invocations) - : VExpr(std::make_shared(), false), - _observed_invocations(observed_invocations) {} - - Status execute_column_impl(VExprContext*, const Block*, const Selector*, size_t count, - ColumnPtr& result_column) const override { - _observed_invocations->push_back(_invocation++); - result_column = ColumnUInt8::create(count, 1); - return Status::OK(); - } - - const std::string& expr_name() const override { return _expr_name; } - bool is_constant() const override { return false; } - bool is_deterministic() const override { return false; } - -private: - std::vector* const _observed_invocations; - mutable int _invocation = 0; - const std::string _expr_name = "StatefulHybridPredicate"; + FileFormat prepared_format = FileFormat::JNI; }; DataTypePtr table_type(const DataTypePtr& type) { @@ -373,8 +332,9 @@ TFileRangeDesc make_paimon_jni_range() { return range; } -TFileRangeDesc make_paimon_range_without_reader_type(TFileFormatType::type format_type) { - TFileRangeDesc range = make_paimon_native_range(format_type); +TFileRangeDesc make_legacy_paimon_native_range(TFileFormatType::type physical_format_type) { + TFileRangeDesc range = make_paimon_native_range(physical_format_type); + range.__set_format_type(TFileFormatType::FORMAT_JNI); range.table_format_params.paimon_params.__isset.reader_type = false; return range; } @@ -714,7 +674,7 @@ TEST(PaimonHybridReaderTest, ClassifiesJniSplitByReaderType) { EXPECT_FALSE(paimon::PaimonHybridReader::TEST_is_jni_split( make_paimon_native_range(TFileFormatType::FORMAT_PARQUET))); EXPECT_FALSE(paimon::PaimonHybridReader::TEST_is_jni_split( - make_paimon_range_without_reader_type(TFileFormatType::FORMAT_JNI))); + make_legacy_paimon_native_range(TFileFormatType::FORMAT_PARQUET))); EXPECT_TRUE(paimon::PaimonHybridReader::TEST_is_jni_split(make_paimon_jni_range())); } @@ -730,114 +690,61 @@ TEST(PaimonHybridReaderTest, ConvertsNativeSplitFileFormat) { .ok()); EXPECT_EQ(file_format, FileFormat::ORC); + ASSERT_TRUE( + paimon::PaimonHybridReader::TEST_to_file_format( + make_legacy_paimon_native_range(TFileFormatType::FORMAT_PARQUET), &file_format) + .ok()); + EXPECT_EQ(file_format, FileFormat::PARQUET); + + ASSERT_TRUE(paimon::PaimonHybridReader::TEST_to_file_format( + make_legacy_paimon_native_range(TFileFormatType::FORMAT_ORC), &file_format) + .ok()); + EXPECT_EQ(file_format, FileFormat::ORC); + auto status = paimon::PaimonHybridReader::TEST_to_file_format(make_paimon_jni_range(), &file_format); EXPECT_FALSE(status.ok()); EXPECT_NE(std::string::npos, status.to_string().find("Unsupported native Paimon file format")); } -TEST(PaimonHybridReaderTest, AdaptiveBatchSizeReachesBothChildReaders) { - paimon::PaimonHybridReader reader; - reader.TEST_install_batch_size_children(); - reader.set_batch_size(321); - const auto child_batch_sizes = reader.TEST_child_batch_sizes(); - EXPECT_EQ(child_batch_sizes.first, 321); - EXPECT_EQ(child_batch_sizes.second, 321); -} - -TEST(PaimonHybridReaderTest, ReportsActiveChildMaterializedBlockStats) { - paimon::PaimonHybridReader reader; - reader.TEST_install_batch_size_children(); - reader._current_split_reader = reader._native_reader.get(); - reader._native_reader->_last_materialized_block_stats = { - .has_materialized_input = true, .rows = 7, .bytes = 70, .allocated_bytes = 96}; - - const auto& stats = reader.last_materialized_block_stats(); - EXPECT_TRUE(stats.has_materialized_input); - EXPECT_EQ(stats.rows, 7); - EXPECT_EQ(stats.bytes, 70); - EXPECT_EQ(stats.allocated_bytes, 96); -} - -TEST(PaimonHybridReaderTest, LateConjunctReachesInitializedNativeAndJniChildren) { +TEST(PaimonHybridReaderTest, NormalizesLegacyNativeSplitFormatBeforeChildPrepare) { RuntimeState state {TQueryOptions(), TQueryGlobals()}; + auto scan_params = make_local_parquet_scan_params(); paimon::PaimonHybridReader reader; + SplitFormatTrackingTableReader* tracking_reader = nullptr; + reader.TEST_set_child_reader_factories( + [&] { + auto child = std::make_unique(); + tracking_reader = child.get(); + return child; + }, + [] { return std::make_unique(); }); ASSERT_TRUE(reader.init({ .projected_columns = {}, .conjuncts = {}, - .format = FileFormat::PARQUET, - .scan_params = nullptr, + .format = FileFormat::JNI, + .scan_params = &scan_params, .io_ctx = nullptr, .runtime_state = &state, .scanner_profile = nullptr, }) .ok()); - auto native_reader = std::make_unique(); - auto jni_reader = std::make_unique(); - auto* native_reader_ptr = native_reader.get(); - auto* jni_reader_ptr = jni_reader.get(); - reader._native_reader = std::move(native_reader); - reader._jni_reader = std::move(jni_reader); - - auto literal = VLiteral::create_shared(std::make_shared(), - Field::create_field(1)); - ASSERT_TRUE(reader.append_conjuncts_with_ownership( - {VExprContext::create_shared(std::move(literal))}, 0) - .ok()); - EXPECT_EQ(native_reader_ptr->appended_conjuncts, 1); - EXPECT_EQ(jni_reader_ptr->appended_conjuncts, 1); - EXPECT_EQ(native_reader_ptr->owned_conjuncts, 0); - EXPECT_EQ(jni_reader_ptr->owned_conjuncts, 0); -} - -TEST(PaimonHybridReaderTest, ScannerStatefulResidualSurvivesNativeJniNativeSwitch) { - RuntimeState state {TQueryOptions(), TQueryGlobals()}; - RuntimeProfile profile("paimon_scanner_stateful_residual"); - auto scan_params = make_local_parquet_scan_params(); - auto hybrid_reader = std::make_unique(); - auto* hybrid_reader_ptr = hybrid_reader.get(); - hybrid_reader_ptr->TEST_set_child_reader_factories( - [] { return std::make_unique(); }, - [] { return std::make_unique(); }); - - std::vector observed_invocations; - auto conjunct = VExprContext::create_shared( - std::make_shared(&observed_invocations)); - ASSERT_TRUE(conjunct->prepare(&state, RowDescriptor {}).ok()); - ASSERT_TRUE(conjunct->open(&state).ok()); - FileScannerV2 scanner(&state, &profile, std::move(hybrid_reader)); - scanner.TEST_set_scanner_conjuncts({std::move(conjunct)}); - const std::vector projected_columns { - make_table_column(0, "id", std::make_shared()), - }; - ASSERT_TRUE(hybrid_reader_ptr - ->init({ - .projected_columns = projected_columns, - .conjuncts = {}, - .format = FileFormat::PARQUET, - .scan_params = &scan_params, - .io_ctx = nullptr, - .runtime_state = &state, - .scanner_profile = &profile, - }) - .ok()); - - auto run_split = [&](FileFormat format, TFileRangeDesc range) { - SplitReadOptions split; - split.current_split_format = format; - split.current_range = std::move(range); - ASSERT_TRUE(hybrid_reader_ptr->prepare_split(split).ok()); - Block block = build_table_block(projected_columns); - bool eos = false; - ASSERT_TRUE(hybrid_reader_ptr->get_block(&block, &eos).ok()); - ASSERT_TRUE(scanner.TEST_filter_output_block(&block).ok()); - }; - run_split(FileFormat::PARQUET, make_paimon_native_range(TFileFormatType::FORMAT_PARQUET)); - run_split(FileFormat::JNI, make_paimon_jni_range()); - run_split(FileFormat::PARQUET, make_paimon_native_range(TFileFormatType::FORMAT_PARQUET)); + SplitReadOptions options; + options.current_range = make_legacy_paimon_native_range(TFileFormatType::FORMAT_PARQUET); + options.current_split_format = FileFormat::JNI; + ASSERT_TRUE(reader.prepare_split(options).ok()); + ASSERT_NE(tracking_reader, nullptr); + EXPECT_EQ(tracking_reader->prepared_format, FileFormat::PARQUET); +} - EXPECT_EQ(observed_invocations, std::vector({0, 1, 2})); +TEST(PaimonHybridReaderTest, AdaptiveBatchSizeReachesBothChildReaders) { + paimon::PaimonHybridReader reader; + reader.TEST_install_batch_size_children(); + reader.set_batch_size(321); + const auto child_batch_sizes = reader.TEST_child_batch_sizes(); + EXPECT_EQ(child_batch_sizes.first, 321); + EXPECT_EQ(child_batch_sizes.second, 321); } TEST(PaimonHybridReaderTest, AggregatesConditionCacheHitsFromBothChildren) { diff --git a/be/test/format_v2/table_reader_test.cpp b/be/test/format_v2/table_reader_test.cpp index d92bbf2b0ac927..bb35fd3926afb0 100644 --- a/be/test/format_v2/table_reader_test.cpp +++ b/be/test/format_v2/table_reader_test.cpp @@ -297,37 +297,6 @@ class NonDeterministicPartitionPredicate final : public VExpr { const std::string _expr_name = "NonDeterministicPartitionPredicate"; }; -class StatefulSequencePredicate final : public VExpr { -public: - explicit StatefulSequencePredicate(std::vector* observed_invocations) - : VExpr(std::make_shared(), false), - _observed_invocations(observed_invocations) {} - - Status execute_column_impl(VExprContext*, const Block*, const Selector*, size_t count, - ColumnPtr& result_column) const override { - DORIS_CHECK(_observed_invocations != nullptr); - _observed_invocations->push_back(_invocation++); - auto result = ColumnUInt8::create(); - result->get_data().resize_fill(count, 1); - result_column = std::move(result); - return Status::OK(); - } - - const std::string& expr_name() const override { return _expr_name; } - bool is_deterministic() const override { return false; } - - Status clone_node(VExprSPtr* cloned_expr) const override { - DORIS_CHECK(cloned_expr != nullptr); - *cloned_expr = std::make_shared(_observed_invocations); - return Status::OK(); - } - -private: - std::vector* const _observed_invocations; - mutable int _invocation = 0; - const std::string _expr_name = "StatefulSequencePredicate"; -}; - class NullableArrayBigintDefaultExpr final : public VExpr { public: explicit NullableArrayBigintDefaultExpr(DataTypePtr data_type) @@ -565,23 +534,6 @@ void write_parquet_file(const std::string& file_path, int32_t id, const std::str builder.build())); } -void write_single_int_parquet_file(const std::string& file_path, const std::string& column_name, - int32_t value) { - auto schema = arrow::schema({arrow::field(column_name, arrow::int32(), false)}); - auto table = arrow::Table::Make(schema, {build_int32_array({value})}); - - auto file_result = arrow::io::FileOutputStream::Open(file_path); - ASSERT_TRUE(file_result.ok()) << file_result.status(); - std::shared_ptr out = *file_result; - - ::parquet::WriterProperties::Builder builder; - builder.version(::parquet::ParquetVersion::PARQUET_2_6); - builder.data_page_version(::parquet::ParquetDataPageVersion::V2); - builder.compression(::parquet::Compression::UNCOMPRESSED); - PARQUET_THROW_NOT_OK(::parquet::arrow::WriteTable(*table, arrow::default_memory_pool(), out, 1, - builder.build())); -} - void write_struct_parquet_file(const std::string& file_path, int32_t id) { auto struct_type = arrow::struct_({arrow::field("id", arrow::int32(), false)}); arrow::StructBuilder builder( @@ -1103,8 +1055,6 @@ struct FakeFileReaderState { bool stop_during_aggregate = false; bool stop_during_read = false; bool not_found_during_init = false; - int batch_count = 1; - int get_block_count = 0; std::shared_ptr last_request; std::optional last_aggregate_request; std::shared_ptr condition_cache_ctx; @@ -1143,7 +1093,7 @@ class FakeFileReader final : public FileReader { RETURN_IF_ERROR(FileReader::open(std::move(request))); _state->last_request = _request; ++_state->open_count; - _returned_batches = 0; + _returned_batch = false; return Status::OK(); } @@ -1152,8 +1102,7 @@ class FakeFileReader final : public FileReader { DORIS_CHECK(rows != nullptr); DORIS_CHECK(eof != nullptr); DORIS_CHECK(_request != nullptr); - ++_state->get_block_count; - if (_returned_batches >= _state->batch_count) { + if (_returned_batch) { *rows = 0; *eof = true; return Status::OK(); @@ -1200,9 +1149,9 @@ class FakeFileReader final : public FileReader { DORIS_CHECK(_state->io_ctx != nullptr); _state->io_ctx->should_stop = true; } - ++_returned_batches; + _returned_batch = true; *rows = 2; - *eof = _state->eof_with_first_batch && _returned_batches >= _state->batch_count; + *eof = _state->eof_with_first_batch; if (_state->condition_cache_ctx != nullptr && !_state->condition_cache_ctx->is_hit && _state->condition_cache_ctx->filter_result != nullptr && !_state->condition_cache_ctx->filter_result->empty()) { @@ -1258,7 +1207,7 @@ class FakeFileReader final : public FileReader { private: std::vector _schema; std::shared_ptr _state; - int _returned_batches = 0; + bool _returned_batch = false; }; class FakeTableReader final : public TableReader { @@ -1354,6 +1303,39 @@ TEST(TableReaderTest, PrepareSplitPrunesPartitionRuntimeFilter) { EXPECT_FALSE(reader.current_split_pruned()); } +TEST(TableReaderTest, PrepareSplitPrunesFileBackedIdentityPartitionRuntimeFilter) { + std::vector projected_columns; + auto identity_partition_source = + make_table_column(0, "part", std::make_shared()); + identity_partition_source.is_partition_key = false; + projected_columns.push_back(std::move(identity_partition_source)); + set_name_identifiers(&projected_columns); + + RuntimeState state {TQueryOptions(), TQueryGlobals()}; + RuntimeProfile profile("scanner"); + TableReader reader; + ASSERT_TRUE(reader.init({ + .projected_columns = projected_columns, + .conjuncts = {}, + .format = FileFormat::PARQUET, + .scan_params = nullptr, + .io_ctx = nullptr, + .runtime_state = &state, + .scanner_profile = &profile, + }) + .ok()); + + SplitReadOptions split; + split.current_range.__set_path("unused-identity-partition-file"); + split.partition_values.emplace("part", Field::create_field(7)); + split.partition_prune_conjuncts.push_back(VExprContext::create_shared( + runtime_filter_wrapper_expr(table_int32_greater_than_expr(0, 0, 10)))); + ASSERT_TRUE(reader.prepare_split(split).ok()); + EXPECT_TRUE(reader.current_split_pruned()); + ASSERT_NE(profile.get_counter("RuntimeFilterPartitionPrunedRangeNum"), nullptr); + EXPECT_EQ(profile.get_counter("RuntimeFilterPartitionPrunedRangeNum")->value(), 1); +} + TEST(TableReaderTest, PrepareSplitDoesNotEvaluateNonDeterministicPartitionPredicate) { std::vector projected_columns; auto partition_column = make_table_column(0, "part", std::make_shared()); @@ -1432,17 +1414,13 @@ TEST(TableReaderTest, ConstantPruningStopsAtUnsafePredicate) { Block block = build_table_block(projected_columns); bool eos = false; ASSERT_TRUE(reader.get_block(&block, &eos).ok()); - EXPECT_TRUE(predicate_executed); + EXPECT_FALSE(predicate_executed); EXPECT_FALSE(eos); - // The file was still opened, proving constant pruning did not jump over the unsafe predicate; - // the predicate is evaluated only after the resulting table row is materialized. EXPECT_EQ(fake_state->open_count, 1); - ASSERT_TRUE(reader.get_block(&block, &eos).ok()); - EXPECT_TRUE(eos); ASSERT_TRUE(reader.close().ok()); } -TEST(TableReaderTest, UnsafePredicateRunsAfterTableMaterialization) { +TEST(TableReaderTest, UnsafePredicateStaysOnScannerPath) { std::vector file_schema; file_schema.push_back(make_file_column(0, "id", std::make_shared())); std::vector projected_columns; @@ -1475,214 +1453,7 @@ TEST(TableReaderTest, UnsafePredicateRunsAfterTableMaterialization) { ASSERT_TRUE(reader.get_block(&block, &eos).ok()); ASSERT_NE(fake_state->last_request, nullptr); EXPECT_TRUE(fake_state->last_request->conjuncts.empty()); - EXPECT_TRUE(predicate_executed); - ASSERT_TRUE(reader.close().ok()); -} - -TEST(TableReaderTest, ScannerOwnedUnsafePredicateIsPassedButNotExecutedByTableReader) { - std::vector file_schema; - file_schema.push_back(make_file_column(0, "id", std::make_shared())); - std::vector projected_columns; - projected_columns.push_back(make_table_column(0, "id", std::make_shared())); - set_name_identifiers(&projected_columns); - - RuntimeState state {TQueryOptions(), TQueryGlobals()}; - bool predicate_executed = false; - auto unsafe_predicate = - std::make_shared(&predicate_executed); - unsafe_predicate->add_child(table_int32_slot_ref(0, 0, "id")); - auto fake_state = std::make_shared(); - FakeTableReader reader(file_schema, fake_state); - ASSERT_TRUE(reader.init({ - .projected_columns = projected_columns, - .conjuncts = {prepared_conjunct(&state, unsafe_predicate)}, - .table_reader_owned_conjunct_count = 0, - .format = FileFormat::PARQUET, - .scan_params = nullptr, - .io_ctx = nullptr, - .runtime_state = &state, - .scanner_profile = nullptr, - }) - .ok()); - - SplitReadOptions split; - split.current_range.__set_path("fake-table-reader-input"); - ASSERT_TRUE(reader.prepare_split(split).ok()); - Block block = build_table_block(projected_columns); - bool eos = false; - ASSERT_TRUE(reader.get_block(&block, &eos).ok()); - - ASSERT_NE(fake_state->last_request, nullptr); - EXPECT_TRUE(fake_state->last_request->conjuncts.empty()); - EXPECT_EQ(reader.TEST_conjunct_count(), 1); - EXPECT_EQ(reader.TEST_table_reader_owned_conjunct_count(), 0); EXPECT_FALSE(predicate_executed); - EXPECT_EQ(block.rows(), 2); - ASSERT_TRUE(reader.close().ok()); -} - -TEST(TableReaderTest, ResidualExpressionStateSurvivesAcrossSplits) { - std::vector file_schema; - file_schema.push_back(make_file_column(0, "id", std::make_shared())); - std::vector projected_columns; - projected_columns.push_back(make_table_column(0, "id", std::make_shared())); - set_name_identifiers(&projected_columns); - - RuntimeState state {TQueryOptions(), TQueryGlobals()}; - std::vector observed_invocations; - auto stateful_predicate = std::make_shared(&observed_invocations); - stateful_predicate->add_child(table_int32_slot_ref(0, 0, "id")); - auto fake_state = std::make_shared(); - FakeTableReader reader(file_schema, fake_state); - ASSERT_TRUE(reader.init({ - .projected_columns = projected_columns, - .conjuncts = {prepared_conjunct(&state, stateful_predicate)}, - .format = FileFormat::PARQUET, - .scan_params = nullptr, - .io_ctx = nullptr, - .runtime_state = &state, - .scanner_profile = nullptr, - }) - .ok()); - - SplitReadOptions split; - split.current_range.__set_path("fake-table-reader-input"); - for (int split_index = 0; split_index < 2; ++split_index) { - ASSERT_TRUE(reader.prepare_split(split).ok()); - Block block = build_table_block(projected_columns); - bool eos = false; - ASSERT_TRUE(reader.get_block(&block, &eos).ok()); - EXPECT_EQ(block.rows(), 2); - ASSERT_TRUE(reader.close().ok()); - } - - EXPECT_EQ(observed_invocations, std::vector({0, 1})); -} - -TEST(TableReaderTest, AllFilteredResidualReturnsAfterOneMaterializedBatch) { - std::vector file_schema; - file_schema.push_back(make_file_column(0, "id", std::make_shared())); - std::vector projected_columns; - projected_columns.push_back(make_table_column(0, "id", std::make_shared())); - set_name_identifiers(&projected_columns); - - RuntimeState state {TQueryOptions(), TQueryGlobals()}; - bool predicate_executed = false; - auto fake_state = std::make_shared(); - fake_state->batch_count = 2; - FakeTableReader reader(file_schema, fake_state); - ASSERT_TRUE(reader.init({ - .projected_columns = projected_columns, - .conjuncts = {prepared_conjunct( - &state, - std::make_shared( - &predicate_executed))}, - .format = FileFormat::PARQUET, - .scan_params = nullptr, - .io_ctx = nullptr, - .runtime_state = &state, - .scanner_profile = nullptr, - }) - .ok()); - - SplitReadOptions split; - split.current_range.__set_path("fake-table-reader-input"); - ASSERT_TRUE(reader.prepare_split(split).ok()); - Block block = build_table_block(projected_columns); - bool eos = false; - ASSERT_TRUE(reader.get_block(&block, &eos).ok()); - - EXPECT_TRUE(predicate_executed); - EXPECT_EQ(block.rows(), 0); - EXPECT_FALSE(eos); - EXPECT_EQ(fake_state->get_block_count, 1); - EXPECT_EQ(fake_state->close_count, 0); - EXPECT_TRUE(reader.last_materialized_block_stats().has_materialized_input); - EXPECT_EQ(reader.last_materialized_block_stats().rows, 2); - EXPECT_GT(reader.last_materialized_block_stats().bytes, 0); - EXPECT_GT(reader.last_materialized_block_stats().allocated_bytes, 0); - ASSERT_TRUE(reader.close().ok()); -} - -TEST(TableReaderTest, LateConjunctFiltersAlreadyOpenSplit) { - std::vector file_schema; - file_schema.push_back(make_file_column(0, "id", std::make_shared())); - std::vector projected_columns; - projected_columns.push_back(make_table_column(0, "id", std::make_shared())); - set_name_identifiers(&projected_columns); - - RuntimeState state {TQueryOptions(), TQueryGlobals()}; - auto fake_state = std::make_shared(); - fake_state->batch_count = 2; - FakeTableReader reader(file_schema, fake_state); - ASSERT_TRUE(reader.init({ - .projected_columns = projected_columns, - .conjuncts = {}, - .format = FileFormat::PARQUET, - .scan_params = nullptr, - .io_ctx = nullptr, - .runtime_state = &state, - .scanner_profile = nullptr, - }) - .ok()); - - SplitReadOptions split; - split.current_range.__set_path("fake-table-reader-input"); - ASSERT_TRUE(reader.prepare_split(split).ok()); - Block block = build_table_block(projected_columns); - bool eos = false; - ASSERT_TRUE(reader.get_block(&block, &eos).ok()); - ASSERT_EQ(block.rows(), 2); - - bool predicate_executed = false; - auto late_predicate = std::make_shared(&predicate_executed); - late_predicate->add_child(table_int32_slot_ref(0, 0, "id")); - ASSERT_TRUE( - reader.append_conjuncts({VExprContext::create_shared(std::move(late_predicate))}).ok()); - block = build_table_block(projected_columns); - ASSERT_TRUE(reader.get_block(&block, &eos).ok()); - EXPECT_TRUE(predicate_executed); - EXPECT_EQ(block.rows(), 0); - EXPECT_EQ(fake_state->get_block_count, 2); - ASSERT_TRUE(reader.close().ok()); -} - -TEST(TableReaderTest, ResidualFilteringHasDedicatedProfileTimer) { - std::vector file_schema; - file_schema.push_back(make_file_column(0, "id", std::make_shared())); - std::vector projected_columns; - projected_columns.push_back(make_table_column(0, "id", std::make_shared())); - set_name_identifiers(&projected_columns); - - RuntimeState state {TQueryOptions(), TQueryGlobals()}; - RuntimeProfile profile("scanner"); - bool predicate_executed = false; - auto fake_state = std::make_shared(); - FakeTableReader reader(file_schema, fake_state); - ASSERT_TRUE(reader.init({ - .projected_columns = projected_columns, - .conjuncts = {prepared_conjunct( - &state, - std::make_shared( - &predicate_executed))}, - .format = FileFormat::PARQUET, - .scan_params = nullptr, - .io_ctx = nullptr, - .runtime_state = &state, - .scanner_profile = &profile, - }) - .ok()); - - SplitReadOptions split; - split.current_range.__set_path("fake-table-reader-input"); - ASSERT_TRUE(reader.prepare_split(split).ok()); - Block block = build_table_block(projected_columns); - bool eos = false; - ASSERT_TRUE(reader.get_block(&block, &eos).ok()); - - EXPECT_TRUE(predicate_executed); - ASSERT_NE(profile.get_counter("ResidualFilterTime"), nullptr); - EXPECT_GT(profile.get_counter("ResidualFilterTime")->value(), 0); ASSERT_TRUE(reader.close().ok()); } @@ -1733,11 +1504,9 @@ TEST(TableReaderTest, ConstantPruningStopsAtUnsafeSlotlessPredicate) { EXPECT_EQ(fake_state->open_count, 1); ASSERT_NE(fake_state->last_request, nullptr); // A slotless unsafe conjunct is an ordering barrier even though it has no TableFilter entry. - // The later predicate must stay on the post-materialization path instead of running inside the + // The later predicate must stay on the scanner's row-level path instead of running inside the // file reader before the unsafe conjunct. EXPECT_TRUE(fake_state->last_request->conjuncts.empty()); - ASSERT_TRUE(reader.get_block(&block, &eos).ok()); - EXPECT_TRUE(eos); ASSERT_TRUE(reader.close().ok()); } @@ -2020,13 +1789,8 @@ TEST(TableReaderTest, SlotlessConjunctDisablesAggregatePushdown) { // presence still prevents the fake aggregate count (3) from replacing the two physical rows. ASSERT_NE(fake_state->last_request, nullptr); EXPECT_TRUE(fake_state->last_request->conjuncts.empty()); - // The two physical rows are then filtered at the table boundary, where slotless predicates are - // evaluated exactly even though they cannot be localized to a file column. - EXPECT_EQ(block.rows(), 0); - EXPECT_FALSE(eos); + EXPECT_EQ(block.rows(), 2); EXPECT_TRUE(predicate_executed); - ASSERT_TRUE(reader.get_block(&block, &eos).ok()); - EXPECT_TRUE(eos); ASSERT_TRUE(reader.close().ok()); } @@ -4575,59 +4339,6 @@ TEST(TableReaderTest, VExprPredicateSurvivesReopenSplit) { std::filesystem::remove_all(test_dir); } -TEST(TableReaderTest, RecomputesPredicateExecutionLayerForEverySplit) { - const auto test_dir = std::filesystem::temp_directory_path() / - "doris_table_reader_split_local_predicate_test"; - std::filesystem::remove_all(test_dir); - std::filesystem::create_directories(test_dir); - - const auto local_file = (test_dir / "local.parquet").string(); - const auto missing_file = (test_dir / "missing.parquet").string(); - write_single_int_parquet_file(local_file, "id", 3); - write_single_int_parquet_file(missing_file, "other", 9); - - std::vector projected_columns; - projected_columns.push_back(make_table_column(0, "id", std::make_shared())); - set_name_identifiers(&projected_columns); - - RuntimeState state {TQueryOptions(), TQueryGlobals()}; - TableReader reader; - ASSERT_TRUE(reader.init({ - .projected_columns = projected_columns, - .conjuncts = {prepared_conjunct( - &state, table_int32_greater_than_expr(0, 0, 2))}, - .format = FileFormat::PARQUET, - .scan_params = nullptr, - .io_ctx = nullptr, - .runtime_state = &state, - .scanner_profile = nullptr, - }) - .ok()); - - ASSERT_TRUE(reader.prepare_split(build_split_options(local_file)).ok()); - Block block = build_table_block(projected_columns); - bool eos = false; - ASSERT_TRUE(reader.get_block(&block, &eos).ok()); - ASSERT_FALSE(eos); - expect_int32_column_values(*block.get_by_position(0).column, {3}); - ASSERT_TRUE(reader.close().ok()); - - // The same predicate cannot be file-local when this split omits `id`. It must be rebuilt as a - // table-level predicate over the materialized NULL instead of inheriting the previous split's - // file-local ownership or escaping without exact evaluation. - ASSERT_TRUE(reader.prepare_split(build_split_options(missing_file)).ok()); - block = build_table_block(projected_columns); - eos = false; - ASSERT_TRUE(reader.get_block(&block, &eos).ok()); - EXPECT_FALSE(eos); - EXPECT_EQ(block.rows(), 0); - ASSERT_TRUE(reader.get_block(&block, &eos).ok()); - EXPECT_TRUE(eos); - - ASSERT_TRUE(reader.close().ok()); - std::filesystem::remove_all(test_dir); -} - TEST(TableReaderTest, CreateScanRequestDeduplicatesSharedPredicateColumns) { const auto int_type = std::make_shared(); const std::vector projected_columns = { diff --git a/be/test/format_v2/wal/wal_reader_test.cpp b/be/test/format_v2/wal/wal_reader_test.cpp new file mode 100644 index 00000000000000..77c14afb12f1ab --- /dev/null +++ b/be/test/format_v2/wal/wal_reader_test.cpp @@ -0,0 +1,171 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +#include "format_v2/wal/wal_reader.h" + +#include + +#include +#include +#include + +#include "agent/be_exec_version_manager.h" +#include "core/block/block.h" +#include "core/column/column_string.h" +#include "core/data_type/data_type_array.h" +#include "core/data_type/data_type_map.h" +#include "core/data_type/data_type_nullable.h" +#include "core/data_type/data_type_number.h" +#include "core/data_type/data_type_string.h" +#include "core/data_type/data_type_struct.h" +#include "io/fs/local_file_system.h" +#include "load/group_commit/wal/wal_file_reader.h" +#include "load/group_commit/wal/wal_writer.h" + +namespace doris::format::wal { +namespace { + +std::string temporary_wal_path() { + const auto root = std::filesystem::temp_directory_path() / + ("doris-wal-v2-" + + std::to_string(std::chrono::steady_clock::now().time_since_epoch().count())); + const auto path = root / "1" / "2" / "1_1_1_test"; + std::filesystem::create_directories(path.parent_path()); + return path.string(); +} + +PBlock serialize_block(const Block& block) { + PBlock pblock; + size_t uncompressed_bytes = 0; + size_t compressed_bytes = 0; + int64_t compress_time = 0; + EXPECT_TRUE(block.serialize(BeExecVersionManager::get_newest_version(), &pblock, + &uncompressed_bytes, &compressed_bytes, &compress_time, + segment_v2::CompressionTypePB::SNAPPY) + .ok()); + return pblock; +} + +} // namespace + +TEST(WalReaderV2Test, ParseColumnIdsPreservesHeaderOrder) { + std::vector column_ids; + ASSERT_TRUE(parse_wal_column_ids("17,4,99", &column_ids).ok()); + EXPECT_EQ(column_ids, (std::vector {17, 4, 99})); +} + +TEST(WalReaderV2Test, ParseColumnIdsRejectsMalformedOrAmbiguousHeaders) { + std::vector column_ids; + EXPECT_FALSE(parse_wal_column_ids("", &column_ids).ok()); + EXPECT_FALSE(parse_wal_column_ids("17,,99", &column_ids).ok()); + EXPECT_FALSE(parse_wal_column_ids("17,nope,99", &column_ids).ok()); + EXPECT_FALSE(parse_wal_column_ids("17,4,17", &column_ids).ok()); +} + +TEST(WalReaderV2Test, WriterBackedReaderPreservesNestedSchemaAndMovesFirstPayload) { + const auto wal_path = temporary_wal_path(); + const auto int_type = make_nullable(std::make_shared()); + const auto array_type = make_nullable(std::make_shared(int_type)); + const auto string_type = make_nullable(std::make_shared()); + const auto map_type = make_nullable(std::make_shared(string_type, int_type)); + const auto struct_type = make_nullable(std::make_shared( + DataTypes {int_type, array_type}, Strings {"id", "nested_items"})); + + Block source; + auto array_column = array_type->create_column(); + array_column->insert_default(); + source.insert({std::move(array_column), array_type, "items"}); + auto string_column = string_type->create_column(); + string_column->insert_data("value", 5); + source.insert({std::move(string_column), string_type, "renamed_later"}); + auto map_column = map_type->create_column(); + map_column->insert_default(); + source.insert({std::move(map_column), map_type, "properties"}); + auto struct_column = struct_type->create_column(); + struct_column->insert_default(); + source.insert({std::move(struct_column), struct_type, "record"}); + auto pblock = serialize_block(source); + + WalWriter writer(wal_path); + ASSERT_TRUE(writer.init(io::global_local_filesystem()).ok()); + ASSERT_TRUE(writer.append_header("17,4,88,99").ok()); + ASSERT_TRUE(writer.append_blocks({&pblock}).ok()); + ASSERT_TRUE(writer.finalize().ok()); + + std::shared_ptr properties; + std::unique_ptr description; + WalReader reader(properties, description, nullptr, nullptr, {}); + reader._wal_reader = std::make_shared(wal_path); + ASSERT_TRUE(reader._wal_reader->init().ok()); + std::string encoded_ids; + ASSERT_TRUE(reader._wal_reader->read_header(reader._version, encoded_ids).ok()); + ASSERT_TRUE(parse_wal_column_ids(encoded_ids, &reader._column_ids).ok()); + + std::vector schema; + ASSERT_TRUE(reader.get_schema(&schema).ok()); + ASSERT_EQ(schema.size(), 4); + ASSERT_EQ(schema[0].children.size(), 1); + EXPECT_TRUE(schema[0].children[0].type->is_nullable()); + ASSERT_EQ(schema[2].children.size(), 2); + EXPECT_TRUE(schema[2].children[0].type->is_nullable()); + EXPECT_TRUE(schema[2].children[1].type->is_nullable()); + ASSERT_EQ(schema[3].children.size(), 2); + EXPECT_EQ(schema[3].children[0].name, "id"); + EXPECT_EQ(schema[3].children[1].name, "nested_items"); + ASSERT_EQ(schema[3].children[1].children.size(), 1); + + auto request = std::make_shared(); + request->local_positions.emplace(LocalColumnId(1), LocalIndex(0)); + request->local_positions.emplace(LocalColumnId(0), LocalIndex(1)); + ASSERT_TRUE(reader.open(std::move(request)).ok()); + Block output({ + {string_type->create_column(), string_type, "renamed"}, + {array_type->create_column(), array_type, "items"}, + }); + size_t rows = 0; + bool eof = false; + ASSERT_TRUE(reader.get_block(&output, &rows, &eof).ok()); + EXPECT_EQ(rows, 1); + EXPECT_FALSE(eof); + EXPECT_EQ(reader._first_block.ByteSizeLong(), 0); + EXPECT_EQ(output.get_by_position(0).column->get_data_at(0).to_string(), "value"); + + ASSERT_TRUE(reader.close().ok()); + std::filesystem::remove_all( + std::filesystem::path(wal_path).parent_path().parent_path().parent_path()); +} + +TEST(WalReaderV2Test, MaterializationTransfersColumnOwnership) { + std::shared_ptr properties; + std::unique_ptr description; + WalReader reader(properties, description, nullptr, nullptr, {}); + reader._request = std::make_shared(); + reader._request->local_positions.emplace(LocalColumnId(0), LocalIndex(0)); + + const auto type = std::make_shared(); + auto source_column = ColumnString::create(); + source_column->insert_data("payload", 7); + const auto* original = source_column.get(); + Block source({{std::move(source_column), type, "value"}}); + Block output({{type->create_column(), type, "value"}}); + + ASSERT_TRUE(reader._materialize_requested_columns(&source, &output).ok()); + EXPECT_FALSE(static_cast(source.get_by_position(0).column)); + EXPECT_EQ(output.get_by_position(0).column.get(), original); +} + +} // namespace doris::format::wal diff --git a/be/test/runtime/runtime_profile_test.cpp b/be/test/runtime/runtime_profile_test.cpp index 7268146e4cdf81..1b31c1b24fc9c9 100644 --- a/be/test/runtime/runtime_profile_test.cpp +++ b/be/test/runtime/runtime_profile_test.cpp @@ -19,8 +19,11 @@ #include +#include #include #include +#include +#include #include "common/exception.h" #include "common/object_pool.h" @@ -558,4 +561,32 @@ TEST(RuntimeProfileTest, TestGetChild) { ASSERT_EQ(child2, root.get_child("Child2")); } +TEST(RuntimeProfileTest, ConcurrentGetOrCreateChildReturnsSingleSharedProfile) { + RuntimeProfile root("Root"); + constexpr size_t kThreadCount = 32; + std::barrier start(kThreadCount); + std::vector children(kThreadCount); + std::vector threads; + threads.reserve(kThreadCount); + + for (size_t i = 0; i < kThreadCount; ++i) { + threads.emplace_back([&, i] { + start.arrive_and_wait(); + children[i] = root.get_or_create_child("SharedChild"); + }); + } + for (auto& thread : threads) { + thread.join(); + } + + ASSERT_NE(children.front(), nullptr); + for (const auto* child : children) { + EXPECT_EQ(child, children.front()); + } + std::vector profile_children; + root.get_children(&profile_children); + ASSERT_EQ(profile_children.size(), 1); + EXPECT_EQ(profile_children.front(), children.front()); +} + } // namespace doris diff --git a/be/test/storage/segment/adaptive_block_size_predictor_test.cpp b/be/test/storage/segment/adaptive_block_size_predictor_test.cpp index 64795dace19a14..60b6f37b8ceeba 100644 --- a/be/test/storage/segment/adaptive_block_size_predictor_test.cpp +++ b/be/test/storage/segment/adaptive_block_size_predictor_test.cpp @@ -89,18 +89,6 @@ TEST_F(AdaptiveBlockSizePredictorTest, NoHistoryReturnsMaxRows) { EXPECT_DOUBLE_EQ(pred.bytes_per_row_for_test(), expected_bpr); } -TEST_F(AdaptiveBlockSizePredictorTest, ExplicitMaterializedSampleUsesPreFilterShape) { - AdaptiveBlockSizePredictor pred(kBlockBytes, 0.0); - - // Callers that filter a block before returning it can still report the rows and bytes that - // were actually materialized upstream. - pred.update(32, 32 * 4096); - - EXPECT_TRUE(pred.has_history_for_test()); - EXPECT_DOUBLE_EQ(pred.bytes_per_row_for_test(), 4096.0); - EXPECT_EQ(pred.predict_next_rows(), 2048); -} - // ── Test 2: EWMA convergence ────────────────────────────────────────────────── // When every update delivers the same sample, the EWMA stays exactly at that // value (0.9*v + 0.1*v == v for any v). diff --git a/docs/file-scanner-v2-parquet-scan-design.md b/docs/file-scanner-v2-parquet-scan-design.md index 70c041a89ccd48..b81a779ac5edcc 100644 --- a/docs/file-scanner-v2-parquet-scan-design.md +++ b/docs/file-scanner-v2-parquet-scan-design.md @@ -333,11 +333,19 @@ flowchart LR B -- "Yes" --> C[Read Dictionary Page] C --> D[Evaluate Predicate on Dictionary Values
Build Dictionary-ID Bitmap] D --> E[Decode Data-page Dictionary IDs
Update SelectionVector Directly] - E --> G[Materialize Only Survivors] + E --> G{Predicate Column Projected?} + G -- "No" --> H[Keep Row Shape Without Typed Values] + G -- "Yes" --> I[Gather Survivors Directly Into Target Column] ``` -- Applies to non-repeated primitive, string-like BYTE_ARRAY / FIXED_LEN_BYTE_ARRAY columns whose - complete Column Chunk uses dictionary data encoding. +- Applies to compatible equality and range predicates on non-repeated primitive columns whose + complete Column Chunk uses `RLE_DICTIONARY` or legacy `PLAIN_DICTIONARY` data encoding. +- Predicate-only batches decode selected IDs straight from the page decoder and never materialize + a row-sized predicate value column. The typed dictionary is cached once per generation. Simple + numeric comparisons build its ID bitmap over contiguous typed values, while string equality and + range comparisons operate directly on dictionary slices. When projection is required, all + fixed-width survivors are written by the filtering loop itself; strings pre-size their character + and offset buffers and copy each compact survivor once. - Safe AND subexpressions may remove components exactly covered by dictionary evaluation. OR or non-equivalent expressions are not rewritten aggressively. - Stateful, potentially throwing, or whole-batch-sensitive expressions disable staged @@ -593,7 +601,9 @@ a dictionary encoding. The predicate is evaluated against the current dictionary entry bitmap; decoded IDs are checked against both dictionary length and bitmap length. A mixed dictionary/plain transition falls back before consuming data. Once selected dictionary reading has advanced a page cursor, loss of dictionary output is corruption rather than a retry through another -path with shifted state. +path with shifted state. Predicate-only slots stop after decoder-level ID filtering and retain only +the output row shape. Projected slots write fixed-width survivors in the filtering loop or gather +compact string survivors directly into pre-sized target buffers. Missing optional indexes or unsupported predicate/type combinations retain rows. Malformed offsets, inconsistent page counts, out-of-range dictionary IDs, overlapping/unsorted invalid ranges, @@ -608,11 +618,11 @@ storage indexes for external Parquet files. | Capability | Granularity | Suitable predicates | Result property | Main limitations | | --- | --- | --- | --- | --- | | Footer Statistics / ZoneMap | Row Group | Ranges, comparisons, IS NULL/IS NOT NULL, and expressions safely convertible to ZoneMap | Can prove the entire group cannot match | Requires valid min/max/null_count and safe type conversion | -| Dictionary Pruning | Row Group | Single-column predicates exactly evaluable over the dictionary domain | Can prove the entire group cannot match | Low-cardinality string-like primitive with complete dictionary encoding | +| Dictionary Pruning | Row Group | Single-column predicates exactly evaluable over the dictionary domain | Can prove the entire group cannot match | Compatible primitive with complete dictionary encoding | | Parquet Bloom Filter | Row Group / Column Chunk | Equality and IN membership-negation predicates | Negative result can prune; positive result requires verification | Controlled by configuration; file must contain Bloom data; false positives are possible | | ColumnIndex | Page | Predicates evaluable from min/max/null | Produces candidate pages and row ranges | Requires an index and decodable compatible types | | OffsetIndex | Page → Row Range | Does not evaluate predicates directly | Maps page results to row numbers and physical skip plans | Normally used with ColumnIndex | -| Dictionary-ID Filter | Row / Batch | Safe single-column string-like predicates | Exact filtering of actual rows | Complete dictionary encoding and non-repeated primitive only | +| Dictionary-ID Filter | Row / Batch | Safe typed equality and range predicates | Exact filtering of actual rows without a full predicate value column | Complete dictionary encoding and non-repeated primitive only | | Condition Cache Bitmap | File-global granule | Stable cacheable conditions | Reuses previous filtering to reduce row ranges | Not a native Parquet index; uncovered ranges remain candidates | ### Index-selection overview @@ -866,7 +876,8 @@ flowchart TD | --- | --- | | Row Group pruning | How many total Row Groups were pruned by Statistics/Dictionary/Bloom, and how much time did each stage take? | | Page index pruning | How many indexes were checked, pages/rows were pruned, ranges selected, and pages skipped? | -| Dictionary row filter | How often were predicates rewritten, dictionaries read, bitmaps built, and attempts successful or rejected? | +| Dictionary row filter | How often were predicates rewritten, dictionaries read, and bitmaps built? `DictFilterTypedCompareColumns` and `DictFilterStringCompareColumns` distinguish typed kernels from the generic expression fallback. | +| Dictionary direct predicate | How many batches and input rows were filtered through dictionary IDs, and how many survivor values were projected? Inspect `DictionaryPredicateDirectBatches/Rows`, `DictionaryPredicateProjectedRows`, and `DictionaryPredicateFusedProjectedRows`. | | Predicate / raw rows | How many rows were read and rejected, and was lazy materialization worthwhile? | | Predicate compaction | Did selection-first evaluation avoid repeated movement? Inspect `PredicateCompactionTime/Bytes/Count`; single-column rounds retain row mappings and compact at multi-column/delete/output boundaries. | | PLAIN direct predicate | How many eligible predicate-only physical batches and input rows bypassed Doris-column materialization? Inspect `PlainPredicateDirectBatches/Rows`. | diff --git a/fe/fe-common/src/main/java/org/apache/doris/catalog/StructField.java b/fe/fe-common/src/main/java/org/apache/doris/catalog/StructField.java index ea8124c8aeb234..e9432c1efad1c5 100644 --- a/fe/fe-common/src/main/java/org/apache/doris/catalog/StructField.java +++ b/fe/fe-common/src/main/java/org/apache/doris/catalog/StructField.java @@ -40,13 +40,22 @@ public class StructField { @SerializedName(value = "containsNull") private final boolean containsNull; // Now always true (nullable field) + // Runtime-only schema change intent; do not persist it as part of the table schema. + private transient boolean commentSpecified; + public static final String DEFAULT_FIELD_NAME = "col"; public StructField(String name, Type type, String comment, boolean containsNull) { + this(name, type, comment, containsNull, !Strings.isNullOrEmpty(comment)); + } + + public StructField(String name, Type type, String comment, boolean containsNull, + boolean commentSpecified) { this.name = name.toLowerCase(); this.type = type; this.comment = comment; this.containsNull = containsNull; + this.commentSpecified = commentSpecified; } public StructField(String name, Type type) { @@ -65,6 +74,10 @@ public String getComment() { return comment; } + public boolean isCommentSpecified() { + return commentSpecified || !Strings.isNullOrEmpty(comment); + } + public String getName() { return name; } @@ -96,7 +109,7 @@ public String toSql(int depth) { if (type != null) { sb.append(":").append(typeSql); } - if (!Strings.isNullOrEmpty(comment)) { + if (isCommentSpecified()) { sb.append(String.format(" comment '%s'", comment)); } return sb.toString(); @@ -116,7 +129,7 @@ public String prettyPrint(int lpad) { typeStr = typeStr.substring(lpad); sb.append(":").append(typeStr); } - if (!Strings.isNullOrEmpty(comment)) { + if (isCommentSpecified()) { sb.append(String.format(" COMMENT '%s'", comment)); } return sb.toString(); @@ -153,7 +166,7 @@ public String toString() { if (type != null) { sb.append(":").append(type); } - if (!Strings.isNullOrEmpty(comment)) { + if (isCommentSpecified()) { sb.append(String.format(" COMMENT '%s'", comment)); } return sb.toString(); diff --git a/fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4 b/fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4 index a710576d1ddf24..0edad842cf7ece 100644 --- a/fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4 +++ b/fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4 @@ -767,11 +767,11 @@ addRollupClause ; alterTableClause - : ADD COLUMN columnDef columnPosition? toRollup? properties=propertyClause? #addColumnClause + : ADD COLUMN columnDefWithPath columnPosition? toRollup? properties=propertyClause? #addColumnClause | ADD COLUMN LEFT_PAREN columnDefs RIGHT_PAREN toRollup? properties=propertyClause? #addColumnsClause - | DROP COLUMN name=identifier fromRollup? properties=propertyClause? #dropColumnClause - | MODIFY COLUMN columnDef columnPosition? fromRollup? + | DROP COLUMN name=qualifiedName fromRollup? properties=propertyClause? #dropColumnClause + | MODIFY COLUMN columnDefWithPath columnPosition? fromRollup? properties=propertyClause? #modifyColumnClause | ORDER BY identifierList fromRollup? properties=propertyClause? #reorderColumnsClause | ADD TEMPORARY? partitionDef @@ -790,14 +790,14 @@ alterTableClause | RENAME newName=identifier #renameClause | RENAME ROLLUP name=identifier newName=identifier #renameRollupClause | RENAME PARTITION name=identifier newName=identifier #renamePartitionClause - | RENAME COLUMN name=identifier newName=identifier #renameColumnClause + | RENAME COLUMN name=qualifiedName TO? newName=identifier #renameColumnClause | ADD indexDef #addIndexClause | DROP INDEX (IF EXISTS)? name=identifier #dropIndexClause | ENABLE FEATURE name=STRING_LITERAL (WITH properties=propertyClause)? #enableFeatureClause | MODIFY DISTRIBUTION (DISTRIBUTED BY (HASH hashKeys=identifierList | RANDOM) (BUCKETS (INTEGER_VALUE | autoBucket=AUTO))?)? #modifyDistributionClause | MODIFY COMMENT comment=STRING_LITERAL #modifyTableCommentClause - | MODIFY COLUMN name=identifier COMMENT comment=STRING_LITERAL #modifyColumnCommentClause + | MODIFY COLUMN name=qualifiedName COMMENT comment=STRING_LITERAL #modifyColumnCommentClause | MODIFY ENGINE TO name=identifier properties=propertyClause? #modifyEngineClause | ADD TEMPORARY? PARTITIONS FROM from=partitionValueList TO to=partitionValueList @@ -1563,6 +1563,17 @@ columnDef (COMMENT comment=STRING_LITERAL)? ; +columnDefWithPath + : columnDef + | colNames+=identifier (DOT colNames+=identifier)+ type=dataType + KEY? + (aggType=aggTypeDef)? + ((GENERATED ALWAYS)? AS LEFT_PAREN generatedExpr=expression RIGHT_PAREN)? + ((NOT)? nullable=NULL)? + (AUTO_INCREMENT (LEFT_PAREN autoIncInitValue=number RIGHT_PAREN)?)? + (COMMENT comment=STRING_LITERAL)? + ; + indexDefs : indexes+=indexDef (COMMA indexes+=indexDef)* ; diff --git a/fe/fe-core/src/main/java/org/apache/doris/alter/Alter.java b/fe/fe-core/src/main/java/org/apache/doris/alter/Alter.java index 89c2d95f4aecce..48900b0121fba5 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/alter/Alter.java +++ b/fe/fe-core/src/main/java/org/apache/doris/alter/Alter.java @@ -418,20 +418,26 @@ private void processAlterTableForExternalTable( table.getDbName(), table.getName(), tableRename.getNewTableName()); } else if (alterClause instanceof AddColumnClause) { AddColumnClause addColumn = (AddColumnClause) alterClause; - table.getCatalog().addColumn(table, addColumn.getColumn(), addColumn.getColPos()); + table.getCatalog().addColumn( + table, addColumn.getColumnPath(), addColumn.getColumn(), addColumn.getColPos()); } else if (alterClause instanceof AddColumnsClause) { AddColumnsClause addColumns = (AddColumnsClause) alterClause; table.getCatalog().addColumns(table, addColumns.getColumns()); } else if (alterClause instanceof DropColumnClause) { DropColumnClause dropColumn = (DropColumnClause) alterClause; - table.getCatalog().dropColumn(table, dropColumn.getColName()); + table.getCatalog().dropColumn(table, dropColumn.getColumnPath()); } else if (alterClause instanceof ColumnRenameClause) { ColumnRenameClause columnRename = (ColumnRenameClause) alterClause; table.getCatalog().renameColumn( - table, columnRename.getColName(), columnRename.getNewColName()); + table, columnRename.getColumnPath(), columnRename.getNewColName()); } else if (alterClause instanceof ModifyColumnClause) { ModifyColumnClause modifyColumn = (ModifyColumnClause) alterClause; - table.getCatalog().modifyColumn(table, modifyColumn.getColumn(), modifyColumn.getColPos()); + table.getCatalog().modifyColumn( + table, modifyColumn.getColumnPath(), modifyColumn.getColumn(), modifyColumn.getColPos()); + } else if (alterClause instanceof ModifyColumnCommentClause) { + ModifyColumnCommentClause modifyColumnComment = (ModifyColumnCommentClause) alterClause; + table.getCatalog().modifyColumnComment( + table, modifyColumnComment.getColumnPath(), modifyColumnComment.getComment()); } else if (alterClause instanceof ReorderColumnsClause) { ReorderColumnsClause reorderColumns = (ReorderColumnsClause) alterClause; table.getCatalog().reorderColumns(table, reorderColumns.getColumnsByPos()); diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/AddColumnClause.java b/fe/fe-core/src/main/java/org/apache/doris/analysis/AddColumnClause.java index 54a98ff9ca1ce4..a59538bae44b48 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/analysis/AddColumnClause.java +++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/AddColumnClause.java @@ -39,6 +39,7 @@ public class AddColumnClause extends AlterTableClause { private static final Logger LOG = LogManager.getLogger(AddColumnClause.class); private ColumnDef columnDef; private String sql; + private ColumnPath columnPath; // Column position private ColumnPosition colPos; // if rollupName is null, add to column to base index. @@ -56,6 +57,10 @@ public ColumnPosition getColPos() { return colPos; } + public ColumnPath getColumnPath() { + return columnPath; + } + public String getRollupName() { return rollupName; } @@ -64,6 +69,7 @@ public AddColumnClause(ColumnDef columnDef, ColumnPosition colPos, String rollup Map properties) { super(AlterOpType.SCHEMA_CHANGE); this.columnDef = columnDef; + this.columnPath = ColumnPath.of(columnDef.getName()); this.colPos = colPos; this.rollupName = rollupName; this.properties = properties; @@ -72,8 +78,16 @@ public AddColumnClause(ColumnDef columnDef, ColumnPosition colPos, String rollup // for nereids public AddColumnClause(String sql, Column column, ColumnPosition colPos, String rollupName, Map properties) { + this(sql, ColumnPath.of(column.getName()), column, colPos, rollupName, properties); + } + + // branch-4.1 dispatches Nereids operations through legacy clauses, so the complete path must + // survive this compatibility bridge instead of being reduced to the nested field's leaf name. + public AddColumnClause(String sql, ColumnPath columnPath, Column column, ColumnPosition colPos, + String rollupName, Map properties) { super(AlterOpType.SCHEMA_CHANGE); this.sql = sql; + this.columnPath = columnPath; this.column = column; this.colPos = colPos; this.rollupName = rollupName; diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/ColumnPath.java b/fe/fe-core/src/main/java/org/apache/doris/analysis/ColumnPath.java new file mode 100644 index 00000000000000..0e28f38de41ef5 --- /dev/null +++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/ColumnPath.java @@ -0,0 +1,92 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.analysis; + +import org.apache.doris.common.util.SqlUtils; + +import com.google.common.base.Preconditions; +import com.google.common.collect.ImmutableList; + +import java.util.Arrays; +import java.util.List; +import java.util.stream.Collectors; + +/** + * Represents a column path used by schema change statements. + */ +public class ColumnPath { + private final ImmutableList parts; + + private ColumnPath(List parts) { + Preconditions.checkArgument(parts != null && !parts.isEmpty(), "column path is empty"); + for (String part : parts) { + Preconditions.checkArgument(part != null && !part.isEmpty(), "column path contains empty part"); + } + this.parts = ImmutableList.copyOf(parts); + } + + public static ColumnPath of(List parts) { + return new ColumnPath(parts); + } + + public static ColumnPath of(String name) { + return new ColumnPath(ImmutableList.of(name)); + } + + public static ColumnPath fromDotName(String name) { + return new ColumnPath(Arrays.asList(name.split("\\."))); + } + + public List getParts() { + return parts; + } + + public boolean isNested() { + return parts.size() > 1; + } + + public String getTopLevelName() { + return parts.get(0); + } + + public String getLeafName() { + return parts.get(parts.size() - 1); + } + + public ColumnPath getParentPath() { + Preconditions.checkState(isNested(), "top-level column path has no parent"); + return new ColumnPath(parts.subList(0, parts.size() - 1)); + } + + public String getParentPathString() { + return getParentPath().getFullPath(); + } + + public String getFullPath() { + return String.join(".", parts); + } + + public String toSql() { + return parts.stream().map(SqlUtils::getIdentSql).collect(Collectors.joining(".")); + } + + @Override + public String toString() { + return getFullPath(); + } +} diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/ColumnPosition.java b/fe/fe-core/src/main/java/org/apache/doris/analysis/ColumnPosition.java index 2e7500969d4e27..fb54828c1ecfea 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/analysis/ColumnPosition.java +++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/ColumnPosition.java @@ -18,6 +18,7 @@ package org.apache.doris.analysis; import org.apache.doris.common.AnalysisException; +import org.apache.doris.common.util.SqlUtils; import com.google.common.base.Strings; @@ -57,7 +58,7 @@ public String toSql() { if (this == FIRST) { sb.append("FIRST"); } else { - sb.append("AFTER `").append(lastCol).append("`"); + sb.append("AFTER ").append(SqlUtils.getIdentSql(lastCol)); } return sb.toString(); } diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/ColumnRenameClause.java b/fe/fe-core/src/main/java/org/apache/doris/analysis/ColumnRenameClause.java index e07d4268d5d26b..dd9b6983312c68 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/analysis/ColumnRenameClause.java +++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/ColumnRenameClause.java @@ -28,11 +28,17 @@ // rename column public class ColumnRenameClause extends AlterTableClause { private String colName; + private ColumnPath columnPath; private String newColName; public ColumnRenameClause(String colName, String newColName) { + this(ColumnPath.of(colName), newColName); + } + + public ColumnRenameClause(ColumnPath columnPath, String newColName) { super(AlterOpType.RENAME); - this.colName = colName; + this.colName = columnPath.getLeafName(); + this.columnPath = columnPath; this.newColName = newColName; this.needTableStable = false; } @@ -41,6 +47,10 @@ public String getColName() { return colName; } + public ColumnPath getColumnPath() { + return columnPath; + } + public String getNewColName() { return newColName; } @@ -75,7 +85,7 @@ public boolean needChangeMTMVState() { @Override public String toSql() { - return "RENAME COLUMN " + colName + " " + newColName; + return "RENAME COLUMN " + columnPath.toSql() + " TO " + newColName; } @Override diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/DropColumnClause.java b/fe/fe-core/src/main/java/org/apache/doris/analysis/DropColumnClause.java index 330d0c9a879c37..33ed0a75dbcfce 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/analysis/DropColumnClause.java +++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/DropColumnClause.java @@ -30,6 +30,7 @@ // Drop one column public class DropColumnClause extends AlterTableClause { private String colName; + private ColumnPath columnPath; private String rollupName; private Map properties; @@ -38,13 +39,22 @@ public String getColName() { return colName; } + public ColumnPath getColumnPath() { + return columnPath; + } + public String getRollupName() { return rollupName; } public DropColumnClause(String colName, String rollupName, Map properties) { + this(ColumnPath.of(colName), rollupName, properties); + } + + public DropColumnClause(ColumnPath columnPath, String rollupName, Map properties) { super(AlterOpType.SCHEMA_CHANGE); - this.colName = colName; + this.colName = columnPath.getLeafName(); + this.columnPath = columnPath; this.rollupName = rollupName; this.properties = properties; } @@ -78,7 +88,7 @@ public boolean needChangeMTMVState() { @Override public String toSql() { StringBuilder sb = new StringBuilder(); - sb.append("DROP COLUMN `").append(colName).append("`"); + sb.append("DROP COLUMN ").append(columnPath.toSql()); if (rollupName != null) { sb.append(" FROM `").append(rollupName).append("`"); } diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/ModifyColumnClause.java b/fe/fe-core/src/main/java/org/apache/doris/analysis/ModifyColumnClause.java index 59b8115855d6da..bdefab45627058 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/analysis/ModifyColumnClause.java +++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/ModifyColumnClause.java @@ -34,6 +34,7 @@ public class ModifyColumnClause extends AlterTableClause { private ColumnDef columnDef; private String sql; + private ColumnPath columnPath; private ColumnPosition colPos; // which rollup is to be modify, if rollup is null, modify base table. private String rollupName; @@ -55,6 +56,10 @@ public ColumnPosition getColPos() { return colPos; } + public ColumnPath getColumnPath() { + return columnPath; + } + public String getRollupName() { return rollupName; } @@ -63,6 +68,7 @@ public ModifyColumnClause(ColumnDef columnDef, ColumnPosition colPos, String rol Map properties) { super(AlterOpType.SCHEMA_CHANGE); this.columnDef = columnDef; + this.columnPath = ColumnPath.of(columnDef.getName()); this.colPos = colPos; this.rollupName = rollup; this.properties = properties; @@ -70,8 +76,14 @@ public ModifyColumnClause(ColumnDef columnDef, ColumnPosition colPos, String rol public ModifyColumnClause(String sql, Column column, ColumnPosition colPos, String rollup, Map properties) { + this(sql, ColumnPath.of(column.getName()), column, colPos, rollup, properties); + } + + public ModifyColumnClause(String sql, ColumnPath columnPath, Column column, ColumnPosition colPos, + String rollup, Map properties) { super(AlterOpType.SCHEMA_CHANGE); this.sql = sql; + this.columnPath = columnPath; this.column = column; this.colPos = colPos; this.rollupName = rollup; diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/ModifyColumnCommentClause.java b/fe/fe-core/src/main/java/org/apache/doris/analysis/ModifyColumnCommentClause.java index 500556866c2c03..ed7b32586ff29c 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/analysis/ModifyColumnCommentClause.java +++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/ModifyColumnCommentClause.java @@ -31,11 +31,17 @@ public class ModifyColumnCommentClause extends AlterTableClause { private static final Logger LOG = LogManager.getLogger(ModifyColumnCommentClause.class); private String colName; + private ColumnPath columnPath; private String comment; public ModifyColumnCommentClause(String colName, String comment) { + this(ColumnPath.of(colName), comment); + } + + public ModifyColumnCommentClause(ColumnPath columnPath, String comment) { super(AlterOpType.MODIFY_COLUMN_COMMENT); - this.colName = colName; + this.colName = columnPath.getLeafName(); + this.columnPath = columnPath; this.comment = Strings.nullToEmpty(comment); } @@ -43,6 +49,10 @@ public String getColName() { return colName; } + public ColumnPath getColumnPath() { + return columnPath; + } + public String getComment() { return comment; } @@ -72,7 +82,7 @@ public boolean needChangeMTMVState() { @Override public String toSql() { StringBuilder sb = new StringBuilder(); - sb.append("MODIFY COLUMN COMMENT ").append(colName); + sb.append("MODIFY COLUMN COMMENT ").append(columnPath.toSql()); sb.append(" '").append(comment).append("'"); return sb.toString(); } diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/Column.java b/fe/fe-core/src/main/java/org/apache/doris/catalog/Column.java index b72194b51b3049..32c66f3e4f69b5 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/catalog/Column.java +++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/Column.java @@ -100,6 +100,10 @@ public class Column implements GsonPostProcessable { private boolean isKey; @SerializedName(value = "isAllowNull") private boolean isAllowNull; + // Runtime-only schema change intent; do not persist it as part of the table schema. + private transient boolean nullableSpecified; + // Runtime-only schema change intent; do not persist it as part of the table schema. + private transient boolean commentSpecified; @SerializedName(value = "isAutoInc") private boolean isAutoInc; @@ -320,6 +324,8 @@ public Column(Column column) { this.isKey = column.isKey(); this.isCompoundKey = column.isCompoundKey(); this.isAllowNull = column.isAllowNull(); + this.nullableSpecified = column.isNullableSpecified(); + this.commentSpecified = column.isCommentSpecified(); this.isAutoInc = column.isAutoInc(); this.defaultValue = column.getDefaultValue(); this.realDefaultValue = column.realDefaultValue; @@ -549,6 +555,14 @@ public boolean isAllowNull() { return isAllowNull; } + public boolean isNullableSpecified() { + return nullableSpecified; + } + + public boolean isCommentSpecified() { + return commentSpecified; + } + public boolean isAutoInc() { return isAutoInc; } @@ -561,6 +575,14 @@ public void setIsAllowNull(boolean isAllowNull) { this.isAllowNull = isAllowNull; } + public void setNullableSpecified(boolean nullableSpecified) { + this.nullableSpecified = nullableSpecified; + } + + public void setCommentSpecified(boolean commentSpecified) { + this.commentSpecified = commentSpecified; + } + public String getDefaultValue() { return this.defaultValue; } diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/ColumnType.java b/fe/fe-core/src/main/java/org/apache/doris/catalog/ColumnType.java index 332cde4898f1e4..776848f0f93fbd 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/catalog/ColumnType.java +++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/ColumnType.java @@ -322,11 +322,10 @@ private static void checkSupportSchemaChangeForComplexType(Type checkType, Type existingNames.add(originalField.getName()); } - // check new field name is not conflict with old field name + // check appended field names do not conflict with existing or earlier appended fields for (int i = originalFields.size(); i < otherStructType.getFields().size(); i++) { - // to check new field name is not conflict with old field name String newFieldName = otherStructType.getFields().get(i).getName(); - if (existingNames.contains(newFieldName)) { + if (!existingNames.add(newFieldName)) { throw new DdlException("Added struct field '" + newFieldName + "' conflicts with existing field"); } } diff --git a/fe/fe-core/src/main/java/org/apache/doris/common/FeNameFormat.java b/fe/fe-core/src/main/java/org/apache/doris/common/FeNameFormat.java index 9917f7c7030279..29f1e948e76239 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/common/FeNameFormat.java +++ b/fe/fe-core/src/main/java/org/apache/doris/common/FeNameFormat.java @@ -108,11 +108,18 @@ public static void checkColumnName(String columnName) throws AnalysisException { } public static void checkColumnNameBypassHiddenColumn(String columnName) throws AnalysisException { + checkColumnNameBypassSystemColumnPrefix(columnName); + checkColumnNamePrefix(columnName, SchemaChangeHandler.SHADOW_NAME_PREFIX); + } + + /** + * Check column name syntax without applying Doris top-level hidden/shadow column prefix rules. + */ + public static void checkColumnNameBypassSystemColumnPrefix(String columnName) throws AnalysisException { if (Strings.isNullOrEmpty(columnName) || !columnName.matches(getColumnNameRegex())) { ErrorReport.reportAnalysisException(ErrorCode.ERR_WRONG_COLUMN_NAME, columnName, getColumnNameRegex()); } - checkColumnNamePrefix(columnName, SchemaChangeHandler.SHADOW_NAME_PREFIX); } private static void checkColumnNamePrefix(String columnName, String prefix) throws AnalysisException { diff --git a/fe/fe-core/src/main/java/org/apache/doris/common/util/PrintableMap.java b/fe/fe-core/src/main/java/org/apache/doris/common/util/PrintableMap.java index 33f87769296b15..d16a9ded5e5b3d 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/common/util/PrintableMap.java +++ b/fe/fe-core/src/main/java/org/apache/doris/common/util/PrintableMap.java @@ -20,6 +20,7 @@ import org.apache.doris.common.maxcompute.MCProperties; import org.apache.doris.datasource.property.metastore.AWSGlueMetaStoreBaseProperties; import org.apache.doris.datasource.property.metastore.AliyunDLFBaseProperties; +import org.apache.doris.datasource.property.metastore.IcebergRestProperties; import org.apache.doris.datasource.property.storage.AzureProperties; import org.apache.doris.datasource.property.storage.COSProperties; import org.apache.doris.datasource.property.storage.GCSProperties; @@ -65,6 +66,9 @@ public class PrintableMap { SENSITIVE_KEY.addAll(ConnectorPropertiesUtils.getSensitiveKeys(S3Properties.class)); SENSITIVE_KEY.addAll(ConnectorPropertiesUtils.getSensitiveKeys(AliyunDLFBaseProperties.class)); SENSITIVE_KEY.addAll(ConnectorPropertiesUtils.getSensitiveKeys(AWSGlueMetaStoreBaseProperties.class)); + // OAuth bearer material must follow the same masking path as storage credentials in + // audit SQL, SHOW CREATE output, and every other printable connector-property surface. + SENSITIVE_KEY.addAll(ConnectorPropertiesUtils.getSensitiveKeys(IcebergRestProperties.class)); SENSITIVE_KEY.addAll(ConnectorPropertiesUtils.getSensitiveKeys(GCSProperties.class)); SENSITIVE_KEY.addAll(ConnectorPropertiesUtils.getSensitiveKeys(AzureProperties.class)); SENSITIVE_KEY.addAll(ConnectorPropertiesUtils.getSensitiveKeys(OSSProperties.class)); diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogIf.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogIf.java index cdc029c909a7ed..6a08bfab63a5ab 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogIf.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogIf.java @@ -17,6 +17,7 @@ package org.apache.doris.datasource; +import org.apache.doris.analysis.ColumnPath; import org.apache.doris.analysis.ColumnPosition; import org.apache.doris.analysis.TableName; import org.apache.doris.catalog.Column; @@ -260,6 +261,15 @@ default void addColumn(TableIf table, Column column, ColumnPosition columnPositi throw new UserException("Not support add column operation"); } + default void addColumn(TableIf table, ColumnPath columnPath, Column column, ColumnPosition columnPosition) + throws UserException { + if (!columnPath.isNested()) { + addColumn(table, column, columnPosition); + return; + } + throw new UserException("Not support nested add column operation"); + } + default void addColumns(TableIf table, List columns) throws UserException { throw new UserException("Not support add columns operation"); } @@ -268,14 +278,47 @@ default void dropColumn(TableIf table, String name) throws UserException { throw new UserException("Not support drop column operation"); } + default void dropColumn(TableIf table, ColumnPath columnPath) throws UserException { + if (!columnPath.isNested()) { + dropColumn(table, columnPath.getTopLevelName()); + return; + } + throw new UserException("Not support nested drop column operation"); + } + default void renameColumn(TableIf table, String oldName, String newName) throws UserException { throw new UserException("Not support rename column operation"); } + default void renameColumn(TableIf table, ColumnPath columnPath, String newName) throws UserException { + if (!columnPath.isNested()) { + renameColumn(table, columnPath.getTopLevelName(), newName); + return; + } + throw new UserException("Not support nested rename column operation"); + } + default void modifyColumn(TableIf table, Column column, ColumnPosition columnPosition) throws UserException { throw new UserException("Not support update column operation"); } + default void modifyColumn(TableIf table, ColumnPath columnPath, Column column, ColumnPosition columnPosition) + throws UserException { + if (!columnPath.isNested()) { + modifyColumn(table, column, columnPosition); + return; + } + throw new UserException("Not support nested modify column operation"); + } + + default void modifyColumnComment(TableIf table, String name, String comment) throws UserException { + modifyColumnComment(table, ColumnPath.of(name), comment); + } + + default void modifyColumnComment(TableIf table, ColumnPath columnPath, String comment) throws UserException { + throw new UserException("Not support modify column comment operation"); + } + default void reorderColumns(TableIf table, List newOrder) throws UserException { throw new UserException("Not support reorder columns operation"); } diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalCatalog.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalCatalog.java index 8ae2d8dd243a64..91e0016c5472b4 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalCatalog.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalCatalog.java @@ -17,6 +17,7 @@ package org.apache.doris.datasource; +import org.apache.doris.analysis.ColumnPath; import org.apache.doris.analysis.ColumnPosition; import org.apache.doris.analysis.TableName; import org.apache.doris.catalog.Column; @@ -1490,6 +1491,12 @@ private void logRefreshExternalTable(ExternalTable dorisTable, long updateTime) @Override public void addColumn(TableIf dorisTable, Column column, ColumnPosition position) throws UserException { + addColumn(dorisTable, ColumnPath.of(column.getName()), column, position); + } + + @Override + public void addColumn(TableIf dorisTable, ColumnPath columnPath, Column column, ColumnPosition position) + throws UserException { makeSureInitialized(); Preconditions.checkState(dorisTable instanceof ExternalTable, dorisTable.getName()); ExternalTable externalTable = (ExternalTable) dorisTable; @@ -1498,11 +1505,11 @@ public void addColumn(TableIf dorisTable, Column column, ColumnPosition position } try { long updateTime = System.currentTimeMillis(); - metadataOps.addColumn(externalTable, column, position, updateTime); + metadataOps.addColumn(externalTable, columnPath, column, position, updateTime); logRefreshExternalTable(externalTable, updateTime); } catch (Exception e) { LOG.warn("Failed to add column {} to table {}.{} in catalog {}", - column.getName(), externalTable.getDbName(), externalTable.getName(), getName(), e); + columnPath.getFullPath(), externalTable.getDbName(), externalTable.getName(), getName(), e); throw e; } } @@ -1528,6 +1535,11 @@ public void addColumns(TableIf dorisTable, List columns) throws UserExce @Override public void dropColumn(TableIf dorisTable, String columnName) throws UserException { + dropColumn(dorisTable, ColumnPath.of(columnName)); + } + + @Override + public void dropColumn(TableIf dorisTable, ColumnPath columnPath) throws UserException { makeSureInitialized(); Preconditions.checkState(dorisTable instanceof ExternalTable, dorisTable.getName()); ExternalTable externalTable = (ExternalTable) dorisTable; @@ -1536,17 +1548,22 @@ public void dropColumn(TableIf dorisTable, String columnName) throws UserExcepti } try { long updateTime = System.currentTimeMillis(); - metadataOps.dropColumn(externalTable, columnName, updateTime); + metadataOps.dropColumn(externalTable, columnPath, updateTime); logRefreshExternalTable(externalTable, updateTime); } catch (Exception e) { LOG.warn("Failed to drop column {} from table {}.{} in catalog {}", - columnName, externalTable.getDbName(), externalTable.getName(), getName(), e); + columnPath.getFullPath(), externalTable.getDbName(), externalTable.getName(), getName(), e); throw e; } } @Override public void renameColumn(TableIf dorisTable, String oldName, String newName) throws UserException { + renameColumn(dorisTable, ColumnPath.of(oldName), newName); + } + + @Override + public void renameColumn(TableIf dorisTable, ColumnPath columnPath, String newName) throws UserException { makeSureInitialized(); Preconditions.checkState(dorisTable instanceof ExternalTable, dorisTable.getName()); ExternalTable externalTable = (ExternalTable) dorisTable; @@ -1555,11 +1572,11 @@ public void renameColumn(TableIf dorisTable, String oldName, String newName) thr } try { long updateTime = System.currentTimeMillis(); - metadataOps.renameColumn(externalTable, oldName, newName, updateTime); + metadataOps.renameColumn(externalTable, columnPath, newName, updateTime); logRefreshExternalTable(externalTable, updateTime); } catch (Exception e) { - LOG.warn("Failed to rename column {} to {} in table {}.{} in catalog {}", - oldName, newName, externalTable.getDbName(), externalTable.getName(), getName(), e); + LOG.warn("Failed to rename column {} to {} in table {}.{} in catalog {}", columnPath.getFullPath(), + newName, externalTable.getDbName(), externalTable.getName(), getName(), e); throw e; } } @@ -1583,6 +1600,45 @@ public void modifyColumn(TableIf dorisTable, Column column, ColumnPosition colum } } + @Override + public void modifyColumn(TableIf dorisTable, ColumnPath columnPath, Column column, ColumnPosition columnPosition) + throws UserException { + makeSureInitialized(); + Preconditions.checkState(dorisTable instanceof ExternalTable, dorisTable.getName()); + ExternalTable externalTable = (ExternalTable) dorisTable; + if (metadataOps == null) { + throw new DdlException("Modify column operation is not supported for catalog: " + getName()); + } + try { + long updateTime = System.currentTimeMillis(); + metadataOps.modifyColumn(externalTable, columnPath, column, columnPosition, updateTime); + logRefreshExternalTable(externalTable, updateTime); + } catch (Exception e) { + LOG.warn("Failed to modify column {} in table {}.{} in catalog {}", + columnPath.getFullPath(), externalTable.getDbName(), externalTable.getName(), getName(), e); + throw e; + } + } + + @Override + public void modifyColumnComment(TableIf dorisTable, ColumnPath columnPath, String comment) throws UserException { + makeSureInitialized(); + Preconditions.checkState(dorisTable instanceof ExternalTable, dorisTable.getName()); + ExternalTable externalTable = (ExternalTable) dorisTable; + if (metadataOps == null) { + throw new DdlException("Modify column comment operation is not supported for catalog: " + getName()); + } + try { + long updateTime = System.currentTimeMillis(); + metadataOps.modifyColumnComment(externalTable, columnPath, comment, updateTime); + logRefreshExternalTable(externalTable, updateTime); + } catch (Exception e) { + LOG.warn("Failed to modify column comment {} in table {}.{} in catalog {}", + columnPath.getFullPath(), externalTable.getDbName(), externalTable.getName(), getName(), e); + throw e; + } + } + @Override public void reorderColumns(TableIf dorisTable, List newOrder) throws UserException { makeSureInitialized(); diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergMetadataOps.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergMetadataOps.java index 04fc1c2865c82b..6520616bf4cee6 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergMetadataOps.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergMetadataOps.java @@ -18,6 +18,7 @@ package org.apache.doris.datasource.iceberg; import org.apache.doris.analysis.AddPartitionFieldClause; +import org.apache.doris.analysis.ColumnPath; import org.apache.doris.analysis.ColumnPosition; import org.apache.doris.analysis.DropPartitionFieldClause; import org.apache.doris.analysis.ReplacePartitionFieldClause; @@ -52,6 +53,7 @@ import com.google.common.annotations.VisibleForTesting; import com.google.common.base.Splitter; +import org.apache.commons.lang3.StringUtils; import org.apache.commons.lang3.exception.ExceptionUtils; import org.apache.iceberg.ManageSnapshots; import org.apache.iceberg.PartitionSpec; @@ -73,6 +75,7 @@ import org.apache.iceberg.expressions.Literal; import org.apache.iceberg.expressions.Term; import org.apache.iceberg.types.Type; +import org.apache.iceberg.types.TypeUtil; import org.apache.iceberg.types.Types; import org.apache.iceberg.types.Types.NestedField; import org.apache.iceberg.view.View; @@ -87,6 +90,8 @@ import java.util.Map; import java.util.Objects; import java.util.Optional; +import java.util.Set; +import java.util.TreeSet; import java.util.concurrent.ThreadPoolExecutor; import java.util.stream.Collectors; import java.util.stream.Stream; @@ -651,19 +656,54 @@ private void addOneColumn(UpdateSchema updateSchema, Column column) throws UserE if (!column.isAllowNull()) { throw new UserException("can't add a non-nullable column to an Iceberg table"); } - org.apache.iceberg.types.Type dorisType = IcebergUtils.dorisTypeToIcebergType(column.getType()); + org.apache.iceberg.types.Type dorisType = + toIcebergTypeForSchemaChange(column.getType(), column.getName()); Literal defaultValue = IcebergUtils.parseIcebergLiteral(column.getDefaultValue(), dorisType); updateSchema.addColumn(column.getName(), dorisType, column.getComment(), defaultValue); } - private void applyPosition(UpdateSchema updateSchema, ColumnPosition position, String columnName) { + private void applyPosition(UpdateSchema updateSchema, ColumnPosition position, ColumnPath columnPath, Schema schema, + String operation) throws UserException { + String columnName = columnPath.getFullPath(); if (position.isFirst()) { updateSchema.moveFirst(columnName); } else { - updateSchema.moveAfter(columnName, position.getLastCol()); + updateSchema.moveAfter(columnName, getPositionReferencePath(schema, columnPath, position, operation)); } } + private void validatePositionTarget(Schema schema, ColumnPath columnPath, String operation) + throws UserException { + if (!columnPath.isNested()) { + return; + } + ResolvedColumnPath parentPath = resolveColumnPath(schema, columnPath.getParentPath(), operation); + if (!parentPath.getType().isStructType()) { + throw new UserException("Cannot apply column position to '" + columnPath.getFullPath() + + "': parent column path '" + parentPath.getFullPath() + "' is not a struct"); + } + } + + @VisibleForTesting + String getPositionReferencePath(ColumnPath columnPath, ColumnPosition position) { + if (position == null || position.isFirst() || !columnPath.isNested()) { + return position == null || position.isFirst() ? null : position.getLastCol(); + } + return columnPath.getParentPathString() + "." + position.getLastCol(); + } + + @VisibleForTesting + String getPositionReferencePath(Schema schema, ColumnPath columnPath, ColumnPosition position, String operation) + throws UserException { + if (position == null || position.isFirst()) { + return null; + } + ColumnPath referencePath = columnPath.isNested() + ? childPath(columnPath.getParentPath(), position.getLastCol()) + : ColumnPath.of(position.getLastCol()); + return resolveColumnPath(schema, referencePath, operation).getFullPath(); + } + private void refreshTable(ExternalTable dorisTable, long updateTime) { Optional> db = dorisCatalog.getDbForReplay(dorisTable.getRemoteDbName()); if (db.isPresent()) { @@ -678,12 +718,16 @@ private void refreshTable(ExternalTable dorisTable, long updateTime) { @Override public void addColumn(ExternalTable dorisTable, Column column, ColumnPosition position, long updateTime) throws UserException { - validateCommonColumnInfo(column); + validateAddColumnMetadata(column, true); Table icebergTable = IcebergUtils.getIcebergTable(dorisTable); + validateRowLineageColumnMutation(icebergTable, column.getName(), "add"); + Schema schema = icebergTable.schema(); + validateNoCaseInsensitiveSiblingCollision( + schema.asStruct(), "", column.getName(), null, "add"); UpdateSchema updateSchema = icebergTable.updateSchema(); addOneColumn(updateSchema, column); if (position != null) { - applyPosition(updateSchema, position, column.getName()); + applyPosition(updateSchema, position, ColumnPath.of(column.getName()), schema, "add"); } try { executionAuthenticator.execute(() -> updateSchema.commit()); @@ -694,12 +738,55 @@ public void addColumn(ExternalTable dorisTable, Column column, ColumnPosition po refreshTable(dorisTable, updateTime); } + @Override + public void addColumn(ExternalTable dorisTable, ColumnPath columnPath, Column column, ColumnPosition position, + long updateTime) throws UserException { + if (!columnPath.isNested()) { + addColumn(dorisTable, column, position, updateTime); + return; + } + validateNestedAddColumnMetadata(column, columnPath); + if (!column.isAllowNull()) { + throw new UserException("New nested field '" + columnPath.getFullPath() + "' must be nullable"); + } + Table icebergTable = IcebergUtils.getIcebergTable(dorisTable); + ResolvedColumnPath parentPath = resolveColumnPath(icebergTable.schema(), columnPath.getParentPath(), "add"); + if (!parentPath.getType().isStructType()) { + throw new UserException("Parent column path '" + columnPath.getParentPathString() + + "' is not a struct in Iceberg table: " + icebergTable.name()); + } + validateNoCaseInsensitiveSiblingCollision(parentPath.getType().asStructType(), + parentPath.getColumnPath(), columnPath.getLeafName(), null, "add"); + + UpdateSchema updateSchema = icebergTable.updateSchema(); + org.apache.iceberg.types.Type dorisType = + toIcebergTypeForSchemaChange(column.getType(), columnPath.getFullPath()); + updateSchema.addColumn(parentPath.getFullPath(), columnPath.getLeafName(), dorisType, + column.getComment()); + if (position != null) { + applyPosition(updateSchema, position, childPath(parentPath.getColumnPath(), columnPath.getLeafName()), + icebergTable.schema(), "add"); + } + try { + executionAuthenticator.execute(() -> updateSchema.commit()); + } catch (Exception e) { + throw new UserException("Failed to add nested column: " + columnPath.getFullPath() + " to table: " + + icebergTable.name() + ", error message is: " + e.getMessage(), e); + } + refreshTable(dorisTable, updateTime); + } + @Override public void addColumns(ExternalTable dorisTable, List columns, long updateTime) throws UserException { Table icebergTable = IcebergUtils.getIcebergTable(dorisTable); + for (Column column : columns) { + validateAddColumnMetadata(column, true); + validateRowLineageColumnMutation(icebergTable, column.getName(), "add"); + } + validateNoCaseInsensitiveTopLevelCollisions(icebergTable.schema(), columns); + UpdateSchema updateSchema = icebergTable.updateSchema(); for (Column column : columns) { - validateCommonColumnInfo(column); addOneColumn(updateSchema, column); } try { @@ -714,8 +801,10 @@ public void addColumns(ExternalTable dorisTable, List columns, long upda @Override public void dropColumn(ExternalTable dorisTable, String columnName, long updateTime) throws UserException { Table icebergTable = IcebergUtils.getIcebergTable(dorisTable); + validateRowLineageColumnMutation(icebergTable, columnName, "drop"); + ResolvedColumnPath columnPath = resolveColumnPath(icebergTable.schema(), ColumnPath.of(columnName), "drop"); UpdateSchema updateSchema = icebergTable.updateSchema(); - updateSchema.deleteColumn(columnName); + updateSchema.deleteColumn(columnPath.getFullPath()); try { executionAuthenticator.execute(() -> updateSchema.commit()); } catch (Exception e) { @@ -725,12 +814,38 @@ public void dropColumn(ExternalTable dorisTable, String columnName, long updateT refreshTable(dorisTable, updateTime); } + @Override + public void dropColumn(ExternalTable dorisTable, ColumnPath columnPath, long updateTime) throws UserException { + if (!columnPath.isNested()) { + dropColumn(dorisTable, columnPath.getTopLevelName(), updateTime); + return; + } + Table icebergTable = IcebergUtils.getIcebergTable(dorisTable); + ResolvedColumnPath resolvedPath = validateNestedStructFieldPath(icebergTable.schema(), columnPath, "drop"); + + UpdateSchema updateSchema = icebergTable.updateSchema(); + updateSchema.deleteColumn(resolvedPath.getFullPath()); + try { + executionAuthenticator.execute(() -> updateSchema.commit()); + } catch (Exception e) { + throw new UserException("Failed to drop nested column: " + columnPath.getFullPath() + " from table: " + + icebergTable.name() + ", error message is: " + e.getMessage(), e); + } + refreshTable(dorisTable, updateTime); + } + @Override public void renameColumn(ExternalTable dorisTable, String oldName, String newName, long updateTime) throws UserException { Table icebergTable = IcebergUtils.getIcebergTable(dorisTable); + validateRowLineageColumnMutation(icebergTable, oldName, "rename"); + validateRowLineageColumnMutation(icebergTable, newName, "rename to"); + Schema schema = icebergTable.schema(); + ResolvedColumnPath oldPath = resolveColumnPath(schema, ColumnPath.of(oldName), "rename"); + validateNoCaseInsensitiveSiblingCollision( + schema.asStruct(), "", newName, oldPath.getField(), "rename"); UpdateSchema updateSchema = icebergTable.updateSchema(); - updateSchema.renameColumn(oldName, newName); + applyRenameColumn(schema, updateSchema, oldPath, newName); try { executionAuthenticator.execute(() -> updateSchema.commit()); } catch (Exception e) { @@ -740,42 +855,113 @@ public void renameColumn(ExternalTable dorisTable, String oldName, String newNam refreshTable(dorisTable, updateTime); } + @Override + public void renameColumn(ExternalTable dorisTable, ColumnPath columnPath, String newName, long updateTime) + throws UserException { + if (!columnPath.isNested()) { + renameColumn(dorisTable, columnPath.getTopLevelName(), newName, updateTime); + return; + } + Table icebergTable = IcebergUtils.getIcebergTable(dorisTable); + ResolvedColumnPath resolvedPath = validateNestedStructFieldPath(icebergTable.schema(), columnPath, "rename"); + ResolvedColumnPath parentPath = resolveColumnPath(icebergTable.schema(), columnPath.getParentPath(), "rename"); + validateNoCaseInsensitiveSiblingCollision(parentPath.getType().asStructType(), + parentPath.getColumnPath(), newName, resolvedPath.getField(), "rename"); + + UpdateSchema updateSchema = icebergTable.updateSchema(); + applyRenameColumn(icebergTable.schema(), updateSchema, resolvedPath, newName); + try { + executionAuthenticator.execute(() -> updateSchema.commit()); + } catch (Exception e) { + throw new UserException("Failed to rename nested column: " + columnPath.getFullPath() + " to " + newName + + " in table: " + icebergTable.name() + ", error message is: " + e.getMessage(), e); + } + refreshTable(dorisTable, updateTime); + } + + private void applyRenameColumn(Schema schema, UpdateSchema updateSchema, + ResolvedColumnPath oldPath, String newName) { + String oldFullPath = oldPath.getFullPath(); + ColumnPath renamedPath = oldPath.getColumnPath().isNested() + ? childPath(oldPath.getColumnPath().getParentPath(), newName) + : ColumnPath.of(newName); + String renamedFullPath = renamedPath.getFullPath(); + boolean identifierFieldRenamed = false; + Set renamedIdentifierFields = new TreeSet<>(); + int renamedFieldId = oldPath.getField().fieldId(); + // Iceberg 1.10.1 does not preserve full identifier paths when an identifier field or one + // of its ancestors is renamed. Use field identity so dotted sibling names are not mistaken for descendants. + for (int identifierFieldId : schema.identifierFieldIds()) { + String identifierField = schema.findColumnName(identifierFieldId); + boolean isRenamedField = identifierFieldId == renamedFieldId; + boolean isDescendant = TypeUtil.ancestorFields(schema, identifierFieldId).stream() + .anyMatch(field -> field.fieldId() == renamedFieldId); + if (isRenamedField || isDescendant) { + renamedIdentifierFields.add(renamedFullPath + identifierField.substring(oldFullPath.length())); + identifierFieldRenamed = true; + } else { + renamedIdentifierFields.add(identifierField); + } + } + + updateSchema.renameColumn(oldFullPath, newName); + if (identifierFieldRenamed) { + updateSchema.setIdentifierFields(renamedIdentifierFields); + } + } + @Override public void modifyColumn(ExternalTable dorisTable, Column column, ColumnPosition position, long updateTime) throws UserException { + // This overload predates nullableSpecified/commentSpecified. Keep its top-level values + // explicit while delegating to the path-aware implementation. + Column explicitColumn = new Column(column); + explicitColumn.setIsKey(false); + explicitColumn.setNullableSpecified(true); + explicitColumn.setCommentSpecified(true); + modifyColumn(dorisTable, ColumnPath.of(column.getName()), explicitColumn, position, updateTime); + } + + private void modifyTopLevelColumn(ExternalTable dorisTable, ColumnPath columnPath, Column column, + ColumnPosition position, long updateTime) throws UserException { Table icebergTable = IcebergUtils.getIcebergTable(dorisTable); - NestedField currentCol = icebergTable.schema().findField(column.getName()); + validateRowLineageColumnMutation(icebergTable, columnPath.getTopLevelName(), "modify"); + NestedField currentCol = icebergTable.schema().asStruct() + .caseInsensitiveField(columnPath.getTopLevelName()); if (currentCol == null) { - throw new UserException("Column " + column.getName() + " does not exist"); + throw new UserException("Column " + columnPath.getTopLevelName() + " does not exist"); } + ResolvedColumnPath resolvedPath = new ResolvedColumnPath(ColumnPath.of(currentCol.name()), + currentCol.type(), currentCol); - validateCommonColumnInfo(column); - UpdateSchema updateSchema = icebergTable.updateSchema(); - + validateModifyColumnMetadata(column, resolvedPath.getFullPath(), true); + org.apache.iceberg.types.Type targetType; if (column.getType().isComplexType()) { - // Complex type processing branch validateForModifyComplexColumn(column, currentCol); - applyComplexTypeChange(updateSchema, column.getName(), currentCol.type(), column.getType()); - if (column.isAllowNull()) { - updateSchema.makeColumnOptional(column.getName()); - } - if (!Objects.equals(currentCol.doc(), column.getComment())) { - updateSchema.updateColumnDoc(column.getName(), column.getComment()); - } + targetType = currentCol.type(); } else { - // Primitive type processing (existing logic) validateForModifyColumn(column, currentCol); - Type icebergType = IcebergUtils.dorisTypeToIcebergType(column.getType()); - updateSchema.updateColumn(column.getName(), icebergType.asPrimitiveType(), column.getComment()); - if (column.isAllowNull()) { - // we can change a required column to optional, but not the other way around - // because we don't know whether there is existing data with null values. - updateSchema.makeColumnOptional(column.getName()); + targetType = resolvePrimitiveTypeForModify( + currentCol.type(), column.getType(), resolvedPath.getFullPath()); + } + + UpdateSchema updateSchema = icebergTable.updateSchema(); + // Preserve the Iceberg doc when MODIFY COLUMN omits COMMENT; only an explicit COMMENT may change it. + String targetComment = resolveTargetComment(currentCol, column); + if (column.getType().isComplexType()) { + applyComplexTypeChange(updateSchema, resolvedPath.getFullPath(), currentCol.type(), + column.getType()); + if (!Objects.equals(currentCol.doc(), targetComment)) { + updateSchema.updateColumnDoc(resolvedPath.getFullPath(), targetComment); } + } else { + applyPrimitiveColumnChange(updateSchema, resolvedPath.getFullPath(), currentCol, + targetType.asPrimitiveType(), targetComment); } + applyExplicitNullableChange(updateSchema, resolvedPath.getFullPath(), column); if (position != null) { - applyPosition(updateSchema, position, column.getName()); + applyPosition(updateSchema, position, resolvedPath.getColumnPath(), icebergTable.schema(), "modify"); } try { executionAuthenticator.execute(() -> updateSchema.commit()); @@ -786,18 +972,158 @@ public void modifyColumn(ExternalTable dorisTable, Column column, ColumnPosition refreshTable(dorisTable, updateTime); } + @Override + public void modifyColumn(ExternalTable dorisTable, ColumnPath columnPath, Column column, ColumnPosition position, + long updateTime) throws UserException { + if (!columnPath.isNested()) { + modifyTopLevelColumn(dorisTable, columnPath, column, position, updateTime); + return; + } + + Table icebergTable = IcebergUtils.getIcebergTable(dorisTable); + ResolvedColumnPath resolvedPath = resolveColumnPath(icebergTable.schema(), columnPath, "modify"); + NestedField currentCol = resolvedPath.getField(); + validateCollectionPseudoFieldComment( + icebergTable.schema(), resolvedPath, column.getComment(), column.isCommentSpecified()); + if (position != null) { + validatePositionTarget(icebergTable.schema(), resolvedPath.getColumnPath(), "modify"); + } + + validateNestedModifyColumnMetadata(column, resolvedPath.getFullPath()); + org.apache.iceberg.types.Type targetType; + if (column.getType().isComplexType()) { + validateForModifyComplexColumn(column, currentCol, columnPath.getFullPath()); + targetType = currentCol.type(); + } else { + validateForModifyColumn(column, currentCol, columnPath.getFullPath()); + targetType = resolvePrimitiveTypeForModify( + currentCol.type(), column.getType(), resolvedPath.getFullPath()); + } + + UpdateSchema updateSchema = icebergTable.updateSchema(); + String targetComment = resolveTargetComment(currentCol, column); + if (column.getType().isComplexType()) { + applyComplexTypeChange(updateSchema, resolvedPath.getFullPath(), currentCol.type(), + column.getType()); + if (!Objects.equals(currentCol.doc(), targetComment)) { + updateSchema.updateColumnDoc(resolvedPath.getFullPath(), targetComment); + } + } else { + applyPrimitiveColumnChange(updateSchema, resolvedPath.getFullPath(), currentCol, + targetType.asPrimitiveType(), targetComment); + } + applyExplicitNullableChange(updateSchema, resolvedPath.getFullPath(), column); + + if (position != null) { + applyPosition(updateSchema, position, resolvedPath.getColumnPath(), icebergTable.schema(), "modify"); + } + + try { + executionAuthenticator.execute(() -> updateSchema.commit()); + } catch (Exception e) { + throw new UserException("Failed to modify nested column: " + columnPath.getFullPath() + " in table: " + + icebergTable.name() + ", error message is: " + e.getMessage(), e); + } + refreshTable(dorisTable, updateTime); + } + + @Override + public void modifyColumnComment(ExternalTable dorisTable, ColumnPath columnPath, String comment, long updateTime) + throws UserException { + Table icebergTable = IcebergUtils.getIcebergTable(dorisTable); + if (!columnPath.isNested()) { + validateRowLineageColumnMutation(icebergTable, columnPath.getTopLevelName(), "modify comment for"); + } + ResolvedColumnPath resolvedPath = resolveColumnPath( + icebergTable.schema(), columnPath, "modify comment"); + validateCollectionPseudoFieldComment(icebergTable.schema(), resolvedPath, comment, true); + + UpdateSchema updateSchema = icebergTable.updateSchema(); + updateSchema.updateColumnDoc(resolvedPath.getFullPath(), StringUtils.defaultString(comment)); + try { + executionAuthenticator.execute(() -> updateSchema.commit()); + } catch (Exception e) { + throw new UserException("Failed to modify column comment: " + columnPath.getFullPath() + " in table: " + + icebergTable.name() + ", error message is: " + e.getMessage(), e); + } + refreshTable(dorisTable, updateTime); + } + + private void validateCollectionPseudoFieldComment(Schema schema, ResolvedColumnPath resolvedPath, + String comment, boolean commentSpecified) throws UserException { + if (!resolvedPath.getColumnPath().isNested() + || (!commentSpecified && StringUtils.isEmpty(comment))) { + return; + } + ResolvedColumnPath parentPath = resolveColumnPath( + schema, resolvedPath.getColumnPath().getParentPath(), "modify comment"); + if (parentPath.getType().isListType() || parentPath.getType().isMapType()) { + throw new UserException("Iceberg does not support comments on collection element or value fields: " + + resolvedPath.getFullPath()); + } + } + + private void applyExplicitNullableChange(UpdateSchema updateSchema, String columnPath, Column column) { + if (column.isNullableSpecified() && column.isAllowNull()) { + updateSchema.makeColumnOptional(columnPath); + } + } + + private String resolveTargetComment(NestedField currentCol, Column column) { + return column.isCommentSpecified() ? column.getComment() : currentCol.doc(); + } + + private org.apache.iceberg.types.Type.PrimitiveType resolvePrimitiveTypeForModify( + org.apache.iceberg.types.Type currentIcebergType, + org.apache.doris.catalog.Type requestedDorisType, String columnPath) throws UserException { + if (isSameMappedDorisType(mappedDorisType(currentIcebergType), requestedDorisType)) { + return currentIcebergType.asPrimitiveType(); + } + org.apache.iceberg.types.Type.PrimitiveType currentType = currentIcebergType.asPrimitiveType(); + org.apache.iceberg.types.Type.PrimitiveType targetType = + toIcebergTypeForSchemaChange(requestedDorisType, columnPath).asPrimitiveType(); + if (!currentType.equals(targetType) && !TypeUtil.isPromotionAllowed(currentType, targetType)) { + throw new UserException("Cannot change column type: " + columnPath + ": " + + currentType + " -> " + targetType); + } + return targetType; + } + + private void applyPrimitiveColumnChange(UpdateSchema updateSchema, String columnPath, + NestedField currentCol, org.apache.iceberg.types.Type.PrimitiveType targetType, + String targetComment) { + if (!currentCol.type().equals(targetType)) { + updateSchema.updateColumn(columnPath, targetType, targetComment); + } else if (!Objects.equals(currentCol.doc(), targetComment)) { + updateSchema.updateColumnDoc(columnPath, targetComment); + } + } + private void validateForModifyColumn(Column column, NestedField currentCol) throws UserException { + validateForModifyColumn(column, currentCol, column.getName()); + } + + private void validateForModifyColumn(Column column, NestedField currentCol, String columnPath) + throws UserException { // check complex type if (column.getType().isComplexType()) { throw new UserException("Modify column type to non-primitive type is not supported: " + column.getType()); } + if (!currentCol.type().isPrimitiveType()) { + throw new UserException("Modify column type from complex to primitive is not supported: " + columnPath); + } // check nullable if (currentCol.isOptional() && !column.isAllowNull()) { - throw new UserException("Can not change nullable column " + column.getName() + " to not null"); + throw new UserException("Can not change nullable column " + columnPath + " to not null"); } } private void validateForModifyComplexColumn(Column column, NestedField currentCol) throws UserException { + validateForModifyComplexColumn(column, currentCol, column.getName()); + } + + private void validateForModifyComplexColumn(Column column, NestedField currentCol, String columnPath) + throws UserException { if (!column.getType().isComplexType()) { throw new UserException("Modify column type to non-complex type is not supported: " + column.getType()); } @@ -807,7 +1133,7 @@ private void validateForModifyComplexColumn(Column column, NestedField currentCo + column.getName()); } - org.apache.doris.catalog.Type oldDorisType = IcebergUtils.icebergTypeToDorisType(oldIcebergType, false, false); + org.apache.doris.catalog.Type oldDorisType = mappedDorisType(oldIcebergType); org.apache.doris.catalog.Type newDorisType = column.getType(); if (!isSameComplexCategory(oldIcebergType, newDorisType)) { throw new UserException("Cannot change complex column type category from " @@ -818,14 +1144,169 @@ private void validateForModifyComplexColumn(Column column, NestedField currentCo } catch (DdlException e) { throw new UserException(e.getMessage(), e); } + validateComplexTypeChanges(oldIcebergType, newDorisType, columnPath); if (currentCol.isOptional() && !column.isAllowNull()) { - throw new UserException("Cannot change nullable column " + column.getName() + " to not null"); + throw new UserException("Cannot change nullable column " + columnPath + " to not null"); } if (column.getDefaultValue() != null || column.getDefaultValueExprDef() != null) { throw new UserException("Complex type default value only supports NULL"); } } + @VisibleForTesting + org.apache.iceberg.types.Type resolveNestedColumnPath(Schema schema, ColumnPath columnPath, String operation) + throws UserException { + return resolveColumnPath(schema, columnPath, operation).getType(); + } + + @VisibleForTesting + String getCanonicalColumnPath(Schema schema, ColumnPath columnPath, String operation) throws UserException { + return resolveColumnPath(schema, columnPath, operation).getFullPath(); + } + + private ResolvedColumnPath resolveColumnPath(Schema schema, ColumnPath columnPath, String operation) + throws UserException { + org.apache.iceberg.types.Type currentType = schema.asStruct(); + NestedField currentField = null; + String currentPath = ""; + List canonicalParts = new ArrayList<>(); + for (String part : columnPath.getParts()) { + if (!currentPath.isEmpty()) { + currentPath += "."; + } + currentPath += part; + + if (currentType.isStructType()) { + NestedField field = currentType.asStructType().caseInsensitiveField(part); + if (field == null) { + throw new UserException("Column path does not exist in Iceberg schema: " + + columnPath.getFullPath()); + } + canonicalParts.add(field.name()); + currentField = field; + currentType = field.type(); + } else if (currentType.isListType()) { + Types.ListType listType = currentType.asListType(); + NestedField elementField = listType.field(listType.elementId()); + if (!elementField.name().equalsIgnoreCase(part)) { + throw new UserException("Expected array element path at '" + currentPath + + "' for Iceberg column path: " + columnPath.getFullPath()); + } + canonicalParts.add(elementField.name()); + currentField = elementField; + currentType = listType.elementType(); + } else if (currentType.isMapType()) { + Types.MapType mapType = currentType.asMapType(); + NestedField keyField = mapType.field(mapType.keyId()); + if (keyField.name().equalsIgnoreCase(part)) { + throw new UserException("Cannot " + operation + " MAP key nested column: " + + columnPath.getFullPath()); + } + NestedField valueField = mapType.field(mapType.valueId()); + if (!valueField.name().equalsIgnoreCase(part)) { + throw new UserException("Expected map value path at '" + currentPath + + "' for Iceberg column path: " + columnPath.getFullPath()); + } + canonicalParts.add(valueField.name()); + currentField = valueField; + currentType = mapType.valueType(); + } else { + throw new UserException("Cannot resolve nested field under primitive column path: " + + columnPath.getFullPath()); + } + } + return new ResolvedColumnPath(ColumnPath.of(canonicalParts), currentType, currentField); + } + + @VisibleForTesting + org.apache.iceberg.types.Type validateNestedStructField(Schema schema, ColumnPath columnPath, String operation) + throws UserException { + return validateNestedStructFieldPath(schema, columnPath, operation).getType(); + } + + private ResolvedColumnPath validateNestedStructFieldPath(Schema schema, ColumnPath columnPath, String operation) + throws UserException { + ResolvedColumnPath parentPath = resolveColumnPath(schema, columnPath.getParentPath(), operation); + org.apache.iceberg.types.Type parentType = parentPath.getType(); + if (!parentType.isStructType()) { + throw new UserException("Parent column path '" + columnPath.getParentPathString() + + "' is not a struct for Iceberg nested " + operation + ": " + columnPath.getFullPath()); + } + NestedField field = parentType.asStructType().caseInsensitiveField(columnPath.getLeafName()); + if (field == null) { + throw new UserException("Column path does not exist in Iceberg schema: " + columnPath.getFullPath()); + } + return new ResolvedColumnPath(childPath(parentPath.getColumnPath(), field.name()), field.type(), field); + } + + private ColumnPath childPath(ColumnPath parentPath, String childName) { + List parts = new ArrayList<>(parentPath.getParts()); + parts.add(childName); + return ColumnPath.of(parts); + } + + @VisibleForTesting + void validateNoCaseInsensitiveSiblingCollision(Types.StructType parentType, ColumnPath parentPath, + String targetName, NestedField sourceField, String operation) throws UserException { + validateNoCaseInsensitiveSiblingCollision( + parentType, parentPath.getFullPath(), targetName, sourceField, operation); + } + + private void validateNoCaseInsensitiveSiblingCollision(Types.StructType parentType, String parentPath, + String targetName, NestedField sourceField, String operation) throws UserException { + NestedField conflictingField = parentType.caseInsensitiveField(targetName); + if (conflictingField != null + && (sourceField == null || conflictingField.fieldId() != sourceField.fieldId())) { + String targetPath = parentPath.isEmpty() ? targetName : parentPath + "." + targetName; + String conflictingPath = parentPath.isEmpty() + ? conflictingField.name() : parentPath + "." + conflictingField.name(); + String columnDescription = parentPath.isEmpty() ? "column" : "nested column"; + throw new UserException("Cannot " + operation + " " + columnDescription + " '" + targetPath + + "': conflicts with existing Iceberg field '" + conflictingPath + "' (case-insensitive)"); + } + } + + private void validateNoCaseInsensitiveTopLevelCollisions(Schema schema, List columns) + throws UserException { + Set requestedNames = new TreeSet<>(String.CASE_INSENSITIVE_ORDER); + for (Column column : columns) { + validateNoCaseInsensitiveSiblingCollision( + schema.asStruct(), "", column.getName(), null, "add"); + if (!requestedNames.add(column.getName())) { + throw new UserException("Cannot add column '" + column.getName() + + "': conflicts with another requested column (case-insensitive)"); + } + } + } + + private static class ResolvedColumnPath { + private final ColumnPath columnPath; + private final org.apache.iceberg.types.Type type; + private final NestedField field; + + private ResolvedColumnPath(ColumnPath columnPath, org.apache.iceberg.types.Type type, NestedField field) { + this.columnPath = columnPath; + this.type = type; + this.field = field; + } + + private ColumnPath getColumnPath() { + return columnPath; + } + + private String getFullPath() { + return columnPath.getFullPath(); + } + + private org.apache.iceberg.types.Type getType() { + return type; + } + + private NestedField getField() { + return field; + } + } + private boolean isSameComplexCategory(Type oldIcebergType, org.apache.doris.catalog.Type newDorisType) { switch (oldIcebergType.typeId()) { case STRUCT: @@ -839,6 +1320,59 @@ private boolean isSameComplexCategory(Type oldIcebergType, org.apache.doris.cata } } + private void validateComplexTypeChanges(org.apache.iceberg.types.Type oldIcebergType, + org.apache.doris.catalog.Type newDorisType, String path) throws UserException { + switch (oldIcebergType.typeId()) { + case STRUCT: + validateStructTypeChanges(oldIcebergType.asStructType(), (StructType) newDorisType, path); + break; + case LIST: + Types.ListType oldListType = oldIcebergType.asListType(); + validateExistingTypeChange(oldListType.elementType(), ((ArrayType) newDorisType).getItemType(), + path + "." + oldListType.field(oldListType.elementId()).name()); + break; + case MAP: + Types.MapType oldMapType = oldIcebergType.asMapType(); + MapType newMapType = (MapType) newDorisType; + org.apache.doris.catalog.Type oldDorisKeyType = mappedDorisType(oldMapType.keyType()); + if (!isSameMappedDorisType(oldDorisKeyType, newMapType.getKeyType())) { + throw new UserException("Cannot change MAP key type from " + + oldDorisKeyType.toSql() + " to " + newMapType.getKeyType().toSql()); + } + validateExistingTypeChange(oldMapType.valueType(), newMapType.getValueType(), + path + "." + oldMapType.field(oldMapType.valueId()).name()); + break; + default: + throw new UserException("Unsupported complex type for modify: " + oldIcebergType); + } + } + + private void validateStructTypeChanges(Types.StructType oldStructType, + StructType newStructType, String path) throws UserException { + List oldFields = oldStructType.fields(); + List newFields = newStructType.getFields(); + for (int i = 0; i < oldFields.size(); i++) { + NestedField oldField = oldFields.get(i); + validateExistingTypeChange(oldField.type(), newFields.get(i).getType(), + path + "." + oldField.name()); + } + for (int i = oldFields.size(); i < newFields.size(); i++) { + StructField newField = newFields.get(i); + toIcebergTypeForSchemaChange(newField.getType(), path + "." + newField.getName()); + } + } + + private void validateExistingTypeChange(org.apache.iceberg.types.Type oldIcebergType, + org.apache.doris.catalog.Type newDorisType, String path) throws UserException { + if (oldIcebergType.isPrimitiveType()) { + if (!isSameMappedDorisType(mappedDorisType(oldIcebergType), newDorisType)) { + toIcebergTypeForSchemaChange(newDorisType, path); + } + return; + } + validateComplexTypeChanges(oldIcebergType, newDorisType, path); + } + private void applyComplexTypeChange(UpdateSchema updateSchema, String path, org.apache.iceberg.types.Type oldIcebergType, org.apache.doris.catalog.Type newDorisType) throws UserException { @@ -858,7 +1392,8 @@ private void applyComplexTypeChange(UpdateSchema updateSchema, String path, } private void applyStructChange(UpdateSchema updateSchema, String path, - Types.StructType oldStructType, StructType newStructType) throws UserException { + Types.StructType oldStructType, StructType newStructType) + throws UserException { List oldFields = oldStructType.fields(); List newFields = newStructType.getFields(); @@ -873,27 +1408,26 @@ private void applyStructChange(UpdateSchema updateSchema, String path, org.apache.iceberg.types.Type oldFieldType = oldField.type(); org.apache.doris.catalog.Type newFieldType = newField.getType(); + String targetComment = newField.isCommentSpecified() + ? newField.getComment() : oldField.doc(); if (oldFieldType.isPrimitiveType()) { - org.apache.doris.catalog.Type oldDorisFieldType = - IcebergUtils.icebergTypeToDorisType(oldFieldType, false, false); - boolean typeChanged = !oldDorisFieldType.equals(newFieldType); - boolean commentChanged = !Objects.equals(oldField.doc(), newField.getComment()); - if (typeChanged || commentChanged) { + org.apache.doris.catalog.Type oldDorisFieldType = mappedDorisType(oldFieldType); + boolean typeChanged = !isSameMappedDorisType(oldDorisFieldType, newFieldType); + boolean commentChanged = !Objects.equals(oldField.doc(), targetComment); + if (typeChanged) { org.apache.iceberg.types.Type newIcebergFieldType = - IcebergUtils.dorisTypeToIcebergType(newFieldType); + toIcebergTypeForSchemaChange(newFieldType, fieldPath); updateSchema.updateColumn(fieldPath, newIcebergFieldType.asPrimitiveType(), - newField.getComment()); + targetComment); + } else if (commentChanged) { + updateSchema.updateColumnDoc(fieldPath, targetComment); } } else { applyComplexTypeChange(updateSchema, fieldPath, oldFieldType, newFieldType); - if (!Objects.equals(oldField.doc(), newField.getComment())) { - updateSchema.updateColumnDoc(fieldPath, newField.getComment()); + if (!Objects.equals(oldField.doc(), targetComment)) { + updateSchema.updateColumnDoc(fieldPath, targetComment); } } - - if (!oldField.isOptional() && newField.getContainsNull()) { - updateSchema.makeColumnOptional(fieldPath); - } } for (int i = oldFields.size(); i < newFields.size(); i++) { @@ -902,13 +1436,14 @@ private void applyStructChange(UpdateSchema updateSchema, String path, throw new UserException("New struct field '" + newField.getName() + "' must be nullable"); } org.apache.iceberg.types.Type newFieldIcebergType = - IcebergUtils.dorisTypeToIcebergType(newField.getType()); + toIcebergTypeForSchemaChange(newField.getType(), path + "." + newField.getName()); updateSchema.addColumn(path, newField.getName(), newFieldIcebergType, newField.getComment()); } } private void applyListChange(UpdateSchema updateSchema, String path, - Types.ListType oldListType, ArrayType newArrayType) throws UserException { + Types.ListType oldListType, ArrayType newArrayType) + throws UserException { String elementPath = path + "." + oldListType.field(oldListType.elementId()).name(); if (oldListType.isElementOptional() && !newArrayType.getContainsNull()) { throw new UserException("Cannot change nullable column " + elementPath + " to not null"); @@ -916,28 +1451,24 @@ private void applyListChange(UpdateSchema updateSchema, String path, org.apache.iceberg.types.Type oldElementType = oldListType.elementType(); org.apache.doris.catalog.Type newElementType = newArrayType.getItemType(); if (oldElementType.isPrimitiveType()) { - org.apache.doris.catalog.Type oldDorisElementType = - IcebergUtils.icebergTypeToDorisType(oldElementType, false, false); - if (!oldDorisElementType.equals(newElementType)) { + org.apache.doris.catalog.Type oldDorisElementType = mappedDorisType(oldElementType); + if (!isSameMappedDorisType(oldDorisElementType, newElementType)) { org.apache.iceberg.types.Type newIcebergElementType = - IcebergUtils.dorisTypeToIcebergType(newElementType); + toIcebergTypeForSchemaChange(newElementType, elementPath); updateSchema.updateColumn(elementPath, newIcebergElementType.asPrimitiveType(), null); } } else { applyComplexTypeChange(updateSchema, elementPath, oldElementType, newElementType); } - if (!oldListType.isElementOptional() && newArrayType.getContainsNull()) { - updateSchema.makeColumnOptional(elementPath); - } } private void applyMapChange(UpdateSchema updateSchema, String path, - Types.MapType oldMapType, MapType newMapType) throws UserException { + Types.MapType oldMapType, MapType newMapType) + throws UserException { org.apache.iceberg.types.Type oldKeyType = oldMapType.keyType(); org.apache.doris.catalog.Type newKeyType = newMapType.getKeyType(); - org.apache.doris.catalog.Type oldDorisKeyType = - IcebergUtils.icebergTypeToDorisType(oldKeyType, false, false); - if (!oldDorisKeyType.equals(newKeyType)) { + org.apache.doris.catalog.Type oldDorisKeyType = mappedDorisType(oldKeyType); + if (!isSameMappedDorisType(oldDorisKeyType, newKeyType)) { throw new UserException("Cannot change MAP key type from " + oldDorisKeyType.toSql() + " to " + newKeyType.toSql()); } @@ -949,22 +1480,29 @@ private void applyMapChange(UpdateSchema updateSchema, String path, org.apache.iceberg.types.Type oldValueType = oldMapType.valueType(); org.apache.doris.catalog.Type newValueType = newMapType.getValueType(); if (oldValueType.isPrimitiveType()) { - org.apache.doris.catalog.Type oldDorisValueType = - IcebergUtils.icebergTypeToDorisType(oldValueType, false, false); - if (!oldDorisValueType.equals(newValueType)) { + org.apache.doris.catalog.Type oldDorisValueType = mappedDorisType(oldValueType); + if (!isSameMappedDorisType(oldDorisValueType, newValueType)) { org.apache.iceberg.types.Type newIcebergValueType = - IcebergUtils.dorisTypeToIcebergType(newValueType); + toIcebergTypeForSchemaChange(newValueType, valuePath); updateSchema.updateColumn(valuePath, newIcebergValueType.asPrimitiveType(), null); } } else { applyComplexTypeChange(updateSchema, valuePath, oldValueType, newValueType); } - if (!oldMapType.isValueOptional() && newMapType.getIsValueContainsNull()) { - updateSchema.makeColumnOptional(valuePath); - } } - private void validateCommonColumnInfo(Column column) throws UserException { + private void validateCommonColumnInfo(Column column, boolean rejectKey) throws UserException { + validateCommonColumnMetadata(column, rejectKey); + toIcebergTypeForSchemaChange(column.getType(), column.getName()); + } + + private void validateCommonColumnMetadata(Column column, boolean rejectKey) throws UserException { + if (rejectKey && column.isKey()) { + throw new UserException("KEY is not supported for Iceberg ADD/MODIFY COLUMN"); + } + if (column.isGeneratedColumn()) { + throw new UserException("Generated columns are not supported for Iceberg ADD/MODIFY COLUMN"); + } // check aggregation method if (column.isAggregated()) { throw new UserException("Can not specify aggregation method for iceberg table column"); @@ -975,16 +1513,87 @@ private void validateCommonColumnInfo(Column column) throws UserException { } } + private org.apache.doris.catalog.Type mappedDorisType(org.apache.iceberg.types.Type icebergType) { + return IcebergUtils.icebergTypeToDorisType(icebergType, + dorisCatalog.getEnableMappingVarbinary(), dorisCatalog.getEnableMappingTimestampTz()); + } + + private boolean isSameMappedDorisType(org.apache.doris.catalog.Type mappedType, + org.apache.doris.catalog.Type requestedType) { + // ScalarType.equals does not compare VARBINARY length, but Iceberg FIXED uses the + // mapped length to distinguish an unchanged view from a requested type change. + return mappedType.equals(requestedType) + && (!mappedType.isVarbinaryType() || mappedType.getLength() == requestedType.getLength()); + } + + private org.apache.iceberg.types.Type toIcebergTypeForSchemaChange( + org.apache.doris.catalog.Type dorisType, String columnPath) throws UserException { + try { + return IcebergUtils.dorisTypeToIcebergType(dorisType); + } catch (UnsupportedOperationException | IllegalArgumentException e) { + throw new UserException("Type " + dorisType.toSql() + + " is not supported for Iceberg column " + columnPath, e); + } + } + + private void validateNestedAddColumnMetadata(Column column, ColumnPath columnPath) throws UserException { + validateCommonColumnInfo(column, true); + if (column.hasDefaultValue() || column.hasOnUpdateDefaultValue()) { + throw new UserException("DEFAULT and ON UPDATE are not supported for Iceberg nested ADD COLUMN: " + + columnPath.getFullPath()); + } + } + + private void validateNestedModifyColumnMetadata(Column column, String columnPath) throws UserException { + validateModifyColumnMetadata(column, columnPath, true); + } + + private void validateAddColumnMetadata(Column column, boolean rejectKey) throws UserException { + validateCommonColumnInfo(column, rejectKey); + if (column.hasOnUpdateDefaultValue()) { + throw new UserException("ON UPDATE is not supported for Iceberg ADD COLUMN: " + column.getName()); + } + } + + private void validateModifyColumnMetadata(Column column, String columnPath, boolean rejectKey) + throws UserException { + validateCommonColumnMetadata(column, rejectKey); + if (column.hasDefaultValue() || column.hasOnUpdateDefaultValue()) { + throw new UserException("Modifying default values is not supported for Iceberg columns: " + columnPath); + } + } + + private void validateRowLineageColumnMutation(Table icebergTable, String columnName, String operation) + throws UserException { + int formatVersion = IcebergUtils.getFormatVersion(icebergTable); + if (formatVersion >= IcebergUtils.ICEBERG_ROW_LINEAGE_MIN_VERSION + && IcebergUtils.isIcebergRowLineageColumn(columnName)) { + throw new UserException("Cannot " + operation + " Iceberg v" + formatVersion + + " reserved row lineage column: " + columnName); + } + } + @Override public void reorderColumns(ExternalTable dorisTable, List newOrder, long updateTime) throws UserException { if (newOrder == null || newOrder.isEmpty()) { throw new UserException("Reorder column failed, new order is empty."); } Table icebergTable = IcebergUtils.getIcebergTable(dorisTable); + List canonicalOrder = new ArrayList<>(newOrder.size()); + Set canonicalNames = new TreeSet<>(String.CASE_INSENSITIVE_ORDER); + for (String columnName : newOrder) { + validateRowLineageColumnMutation(icebergTable, columnName, "reorder"); + String canonicalName = resolveColumnPath( + icebergTable.schema(), ColumnPath.of(columnName), "reorder").getFullPath(); + if (!canonicalNames.add(canonicalName)) { + throw new UserException("Duplicate column in reorder columns: " + columnName); + } + canonicalOrder.add(canonicalName); + } UpdateSchema updateSchema = icebergTable.updateSchema(); - updateSchema.moveFirst(newOrder.get(0)); - for (int i = 1; i < newOrder.size(); i++) { - updateSchema.moveAfter(newOrder.get(i), newOrder.get(i - 1)); + updateSchema.moveFirst(canonicalOrder.get(0)); + for (int i = 1; i < canonicalOrder.size(); i++) { + updateSchema.moveAfter(canonicalOrder.get(i), canonicalOrder.get(i - 1)); } try { executionAuthenticator.execute(() -> updateSchema.commit()); diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/source/IcebergScanNode.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/source/IcebergScanNode.java index d4167d7bc8aa40..d5ee31ace104d7 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/source/IcebergScanNode.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/source/IcebergScanNode.java @@ -956,13 +956,12 @@ private Split createIcebergSplit(FileScanTask fileScanTask) { } if (sessionVariable.isEnableRuntimeFilterPartitionPrune()) { Map partitionInfoMap = partitionMapInfos.computeIfAbsent( - partitionData, k -> { - return IcebergUtils.getPartitionInfoMap(partitionData, partitionSpec, - sessionVariable.getTimeZone()); - }); - // Only set partition values if all partitions are identity transform - // For non-identity partitions, getPartitionInfoMap returns null to skip dynamic partition pruning - if (partitionInfoMap != null) { + partitionData, k -> IcebergUtils.getIdentityPartitionInfoMap( + partitionData, partitionSpec, icebergTable, sessionVariable.getTimeZone())); + // A spec may mix identity and transformed fields. Keep its identity values so a + // runtime filter can prune that split without treating source columns as constants + // for files written under another evolved spec. + if (!partitionInfoMap.isEmpty()) { split.setIcebergPartitionValues(partitionInfoMap); } } else { diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/operations/ExternalMetadataOps.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/operations/ExternalMetadataOps.java index 3407e8e7cec09b..13b3d9fea41c54 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/operations/ExternalMetadataOps.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/operations/ExternalMetadataOps.java @@ -17,6 +17,7 @@ package org.apache.doris.datasource.operations; +import org.apache.doris.analysis.ColumnPath; import org.apache.doris.analysis.ColumnPosition; import org.apache.doris.catalog.Column; import org.apache.doris.common.DdlException; @@ -228,6 +229,15 @@ default void addColumn(ExternalTable dorisTable, Column column, ColumnPosition p throw new UnsupportedOperationException("Add column operation is not supported for this table type."); } + default void addColumn(ExternalTable dorisTable, ColumnPath columnPath, Column column, ColumnPosition position, + long updateTime) throws UserException { + if (!columnPath.isNested()) { + addColumn(dorisTable, column, position, updateTime); + return; + } + throw new UnsupportedOperationException("Nested add column operation is not supported for this table type."); + } + /** * add columns for external table * @@ -252,6 +262,15 @@ default void dropColumn(ExternalTable dorisTable, String columnName, long update throw new UnsupportedOperationException("Drop column operation is not supported for this table type."); } + default void dropColumn(ExternalTable dorisTable, ColumnPath columnPath, long updateTime) + throws UserException { + if (!columnPath.isNested()) { + dropColumn(dorisTable, columnPath.getTopLevelName(), updateTime); + return; + } + throw new UnsupportedOperationException("Nested drop column operation is not supported for this table type."); + } + /** * rename column for external table * @@ -265,6 +284,15 @@ default void renameColumn(ExternalTable dorisTable, String oldName, String newNa throw new UnsupportedOperationException("Rename column operation is not supported for this table type."); } + default void renameColumn(ExternalTable dorisTable, ColumnPath columnPath, String newName, long updateTime) + throws UserException { + if (!columnPath.isNested()) { + renameColumn(dorisTable, columnPath.getTopLevelName(), newName, updateTime); + return; + } + throw new UnsupportedOperationException("Nested rename column operation is not supported for this table type."); + } + /** * update column for external table * @@ -278,6 +306,30 @@ default void modifyColumn(ExternalTable dorisTable, Column column, ColumnPositio throw new UnsupportedOperationException("Modify column operation is not supported for this table type."); } + default void modifyColumn(ExternalTable dorisTable, ColumnPath columnPath, Column column, ColumnPosition position, + long updateTime) throws UserException { + if (!columnPath.isNested()) { + modifyColumn(dorisTable, column, position, updateTime); + return; + } + throw new UnsupportedOperationException("Nested modify column operation is not supported for this table type."); + } + + /** + * modify column comment for external table + * + * @param dorisTable + * @param columnPath + * @param comment + * @param updateTime + * @throws UserException + */ + default void modifyColumnComment(ExternalTable dorisTable, ColumnPath columnPath, String comment, long updateTime) + throws UserException { + throw new UnsupportedOperationException( + "Modify column comment operation is not supported for this table type."); + } + /** * reorder columns for external table * diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/source/PaimonScanNode.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/source/PaimonScanNode.java index 067f27c664d784..aab64aeb034079 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/source/PaimonScanNode.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/source/PaimonScanNode.java @@ -302,21 +302,11 @@ private void setPaimonParams(TFileRangeDesc rangeDesc, PaimonSplit paimonSplit) String fileFormat = getFileFormat(paimonSplit.getPathString()); if (split != null) { - // use jni reader or paimon-cpp reader rangeDesc.setFormatType(TFileFormatType.FORMAT_JNI); - // Use Paimon native serialization for paimon-cpp reader - if (sessionVariable.isEnablePaimonCppReader() && split instanceof DataSplit) { - fileDesc.setReaderType(TPaimonReaderType.PAIMON_CPP); - fileDesc.setPaimonSplit(PaimonUtil.encodeDataSplitToString((DataSplit) split)); - } else { - fileDesc.setReaderType(TPaimonReaderType.PAIMON_JNI); - fileDesc.setPaimonSplit(PaimonUtil.encodeObjectToString(split)); - } - // Set table location for paimon-cpp reader - String tableLocation = source.getTableLocation(); - if (tableLocation != null) { - fileDesc.setPaimonTable(tableLocation); - } + // A logical DataSplit may span multiple files, so keep it intact for the JNI reader + // until the C++ path has a split-aware V2 adapter. + fileDesc.setReaderType(TPaimonReaderType.PAIMON_JNI); + fileDesc.setPaimonSplit(PaimonUtil.encodeObjectToString(split)); rangeDesc.setSelfSplitWeight(paimonSplit.getSelfSplitWeight()); } else { // use native reader diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/property/metastore/IcebergRestProperties.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/property/metastore/IcebergRestProperties.java index f457c90ff40868..3c3da3fce5f2ee 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/property/metastore/IcebergRestProperties.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/property/metastore/IcebergRestProperties.java @@ -81,6 +81,7 @@ public class IcebergRestProperties extends AbstractIcebergProperties { @ConnectorProperty(names = {"iceberg.rest.oauth2.token"}, required = false, + sensitive = true, description = "The oauth2 token for the iceberg rest catalog service.") private String icebergRestOauth2Token; diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java index f44fe31afde594..25f58441f8c35c 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java @@ -22,6 +22,7 @@ import org.apache.doris.analysis.ArithmeticExpr.Operator; import org.apache.doris.analysis.BrokerDesc; import org.apache.doris.analysis.ColumnNullableType; +import org.apache.doris.analysis.ColumnPath; import org.apache.doris.analysis.ColumnPosition; import org.apache.doris.analysis.DbName; import org.apache.doris.analysis.EncryptKeyName; @@ -142,6 +143,7 @@ import org.apache.doris.nereids.DorisParser.CleanLabelContext; import org.apache.doris.nereids.DorisParser.CollateContext; import org.apache.doris.nereids.DorisParser.ColumnDefContext; +import org.apache.doris.nereids.DorisParser.ColumnDefWithPathContext; import org.apache.doris.nereids.DorisParser.ColumnDefsContext; import org.apache.doris.nereids.DorisParser.ColumnReferenceContext; import org.apache.doris.nereids.DorisParser.CommentRelationHintContext; @@ -1099,6 +1101,7 @@ import org.apache.doris.nereids.types.coercion.CharacterType; import org.apache.doris.nereids.util.ExpressionUtils; import org.apache.doris.nereids.util.RelationUtil; +import org.apache.doris.nereids.util.SqlLiteralUtils; import org.apache.doris.nereids.util.Utils; import org.apache.doris.policy.FilterType; import org.apache.doris.policy.PolicyTypeEnum; @@ -1160,6 +1163,16 @@ public class LogicalPlanBuilder extends DorisParserBaseVisitor { private static String DEFAULT_NESTED_COLUMN_NAME = "unnest"; private static String DEFAULT_ORDINALITY_COLUMN_NAME = "ordinality"; + private static class ColumnDefinitionWithPath { + private final ColumnDefinition columnDefinition; + private final ColumnPath columnPath; + + private ColumnDefinitionWithPath(ColumnDefinition columnDefinition, ColumnPath columnPath) { + this.columnDefinition = columnDefinition; + this.columnPath = columnPath; + } + } + // Sort the parameters with token position to keep the order with original placeholders // in prepared statement.Otherwise, the order maybe broken private final Map tokenPosToParameters = Maps.newTreeMap((pos1, pos2) -> { @@ -1180,6 +1193,18 @@ public LogicalPlanBuilder(Map selectHintMap) { this.selectHintMap = selectHintMap; } + private static String requireNonEmptyColumnIdentifier(ParserRuleContext ctx, String identifier) { + if (identifier.isEmpty()) { + throw new ParseException("Quoted identifier cannot be empty", ctx); + } + return identifier; + } + + private static ColumnPath parseColumnPath(ParserRuleContext ctx, List parts) { + parts.forEach(part -> requireNonEmptyColumnIdentifier(ctx, part)); + return ColumnPath.of(parts); + } + @SuppressWarnings("unchecked") protected T typedVisit(ParseTree ctx) { return (T) ctx.accept(this); @@ -3723,18 +3748,7 @@ public Literal visitIntegerLiteral(IntegerLiteralContext ctx) { @Override public Literal visitStringLiteral(StringLiteralContext ctx) { - String txt = ctx.STRING_LITERAL().getText(); - String s = txt.substring(1, txt.length() - 1); - if (txt.charAt(0) == '\'') { - // for single quote string, '' should be converted to ' - s = s.replace("''", "'"); - } else if (txt.charAt(0) == '"') { - // for double quote string, "" should be converted to " - s = s.replace("\"\"", "\""); - } - if (!SqlModeHelper.hasNoBackSlashEscapes()) { - s = LogicalPlanBuilderAssistant.escapeBackSlash(s); - } + String s = SqlLiteralUtils.parseStringLiteral(ctx.STRING_LITERAL().getText()); int strLength = Utils.containChinese(s) ? s.length() * StringLikeLiteral.CHINESE_CHAR_BYTE_LENGTH : s.length(); if (strLength > ScalarType.MAX_VARCHAR_LENGTH) { return new StringLiteral(s); @@ -3901,6 +3915,13 @@ public List visitIdentifierSeq(IdentifierSeqContext ctx) { .collect(ImmutableList.toImmutableList()); } + @Override + public List visitQualifiedName(QualifiedNameContext ctx) { + return ctx.identifier().stream() + .map(RuleContext::getText) + .collect(ImmutableList.toImmutableList()); + } + @Override public EqualTo visitUpdateAssignment(UpdateAssignmentContext ctx) { return new EqualTo(new UnboundSlot( @@ -4189,9 +4210,8 @@ public ColumnDefinition visitColumnDef(ColumnDefContext ctx) { e.getCause()); } } - //comment should remove '\' and '(") at the beginning and end - String comment = ctx.comment != null ? ctx.comment.getText().substring(1, ctx.comment.getText().length() - 1) - .replace("\\", "") : ""; + String comment = ctx.comment != null + ? SqlLiteralUtils.parseStringLiteral(ctx.comment.getText()) : ""; long autoIncInitValue = -1; if (ctx.AUTO_INCREMENT() != null) { if (ctx.autoIncInitValue != null) { @@ -4209,7 +4229,65 @@ public ColumnDefinition visitColumnDef(ColumnDefContext ctx) { ? Optional.of(new GeneratedColumnDesc(ctx.generatedExpr.getText(), getExpression(ctx.generatedExpr))) : Optional.empty(); return new ColumnDefinition(colName, colType, isKey, aggType, nullableType, autoIncInitValue, defaultValue, - onUpdateDefaultValue, comment, desc); + onUpdateDefaultValue, comment, ctx.comment != null, true, desc); + } + + @Override + public ColumnDefinitionWithPath visitColumnDefWithPath(ColumnDefWithPathContext ctx) { + if (ctx.columnDef() != null) { + ColumnDefinition columnDefinition = visitColumnDef(ctx.columnDef()); + ColumnPath columnPath = parseColumnPath(ctx, Collections.singletonList(columnDefinition.getName())); + return new ColumnDefinitionWithPath(columnDefinition, columnPath); + } + + ColumnPath columnPath = parseColumnPath(ctx, ctx.colNames.stream() + .map(RuleContext::getText) + .collect(Collectors.toList())); + String colName = columnPath.getLeafName(); + DataType colType = ctx.type instanceof PrimitiveDataTypeContext + ? visitPrimitiveDataType(((PrimitiveDataTypeContext) ctx.type)) + : ctx.type instanceof ComplexDataTypeContext + ? visitComplexDataType((ComplexDataTypeContext) ctx.type) + : ctx.type instanceof VariantPredefinedFieldsContext + ? visitVariantPredefinedFields((VariantPredefinedFieldsContext) ctx.type) + : visitAggStateDataType((AggStateDataTypeContext) ctx.type); + colType = colType.conversion(); + boolean isKey = ctx.KEY() != null; + ColumnNullableType nullableType = ColumnNullableType.DEFAULT; + if (ctx.NOT() != null) { + nullableType = ColumnNullableType.NOT_NULLABLE; + } else if (ctx.nullable != null) { + nullableType = ColumnNullableType.NULLABLE; + } + String aggTypeString = ctx.aggType != null ? ctx.aggType.getText() : null; + AggregateType aggType = null; + if (aggTypeString != null) { + try { + aggType = AggregateType.valueOf(aggTypeString.toUpperCase()); + } catch (Exception e) { + throw new AnalysisException(String.format("Aggregate type %s is unsupported", aggTypeString), + e.getCause()); + } + } + String comment = ctx.comment != null + ? SqlLiteralUtils.parseStringLiteral(ctx.comment.getText()) : ""; + long autoIncInitValue = -1; + if (ctx.AUTO_INCREMENT() != null) { + if (ctx.autoIncInitValue != null) { + autoIncInitValue = Long.valueOf(ctx.autoIncInitValue.getText()); + if (autoIncInitValue < 0) { + throw new AnalysisException("AUTO_INCREMENT start value can not be negative."); + } + } else { + autoIncInitValue = Long.valueOf(1); + } + } + Optional desc = ctx.generatedExpr != null + ? Optional.of(new GeneratedColumnDesc(ctx.generatedExpr.getText(), getExpression(ctx.generatedExpr))) + : Optional.empty(); + ColumnDefinition columnDefinition = new ColumnDefinition(colName, colType, isKey, aggType, nullableType, + autoIncInitValue, Optional.empty(), Optional.empty(), comment, ctx.comment != null, true, desc); + return new ColumnDefinitionWithPath(columnDefinition, columnPath); } @Override @@ -5396,14 +5474,11 @@ public List visitComplexColTypeList(ComplexColTypeListContext ctx) @Override public StructField visitComplexColType(ComplexColTypeContext ctx) { - String comment; - if (ctx.commentSpec() != null) { - comment = ctx.commentSpec().STRING_LITERAL().getText(); - comment = LogicalPlanBuilderAssistant.escapeBackSlash(comment.substring(1, comment.length() - 1)); - } else { - comment = ""; - } - return new StructField(ctx.identifier().getText(), typedVisit(ctx.dataType()), true, comment); + String comment = ctx.commentSpec() == null ? "" + : SqlLiteralUtils.parseStringLiteral( + ctx.commentSpec().STRING_LITERAL().getText()); + return new StructField(ctx.identifier().getText(), typedVisit(ctx.dataType()), true, + comment, ctx.commentSpec() != null); } private String parseConstant(ConstantContext context) { @@ -6130,7 +6205,7 @@ public AlterTableCommand visitAlterTableProperties(DorisParser.AlterTablePropert @Override public AlterTableOp visitAddColumnClause(AddColumnClauseContext ctx) { - ColumnDefinition columnDefinition = visitColumnDef(ctx.columnDef()); + ColumnDefinitionWithPath columnDefinitionWithPath = visitColumnDefWithPath(ctx.columnDefWithPath()); ColumnPosition columnPosition = null; if (ctx.columnPosition() != null) { if (ctx.columnPosition().FIRST() != null) { @@ -6143,7 +6218,8 @@ public AlterTableOp visitAddColumnClause(AddColumnClauseContext ctx) { Map properties = ctx.properties != null ? Maps.newHashMap(visitPropertyClause(ctx.properties)) : Maps.newHashMap(); - return new AddColumnOp(columnDefinition, columnPosition, rollupName, properties); + return new AddColumnOp(columnDefinitionWithPath.columnDefinition, columnDefinitionWithPath.columnPath, + columnPosition, rollupName, properties); } @Override @@ -6158,17 +6234,17 @@ public AlterTableOp visitAddColumnsClause(AddColumnsClauseContext ctx) { @Override public AlterTableOp visitDropColumnClause(DropColumnClauseContext ctx) { - String columnName = ctx.name.getText(); + ColumnPath columnPath = parseColumnPath(ctx.name, visitQualifiedName(ctx.name)); String rollupName = ctx.fromRollup() != null ? ctx.fromRollup().rollup.getText() : null; Map properties = ctx.properties != null ? Maps.newHashMap(visitPropertyClause(ctx.properties)) : Maps.newHashMap(); - return new DropColumnOp(columnName, rollupName, properties); + return new DropColumnOp(columnPath, rollupName, properties); } @Override public AlterTableOp visitModifyColumnClause(ModifyColumnClauseContext ctx) { - ColumnDefinition columnDefinition = visitColumnDef(ctx.columnDef()); + ColumnDefinitionWithPath columnDefinitionWithPath = visitColumnDefWithPath(ctx.columnDefWithPath()); ColumnPosition columnPosition = null; if (ctx.columnPosition() != null) { if (ctx.columnPosition().FIRST() != null) { @@ -6181,12 +6257,14 @@ public AlterTableOp visitModifyColumnClause(ModifyColumnClauseContext ctx) { Map properties = ctx.properties != null ? Maps.newHashMap(visitPropertyClause(ctx.properties)) : Maps.newHashMap(); - return new ModifyColumnOp(columnDefinition, columnPosition, rollupName, properties); + return new ModifyColumnOp(columnDefinitionWithPath.columnDefinition, columnDefinitionWithPath.columnPath, + columnPosition, rollupName, properties); } @Override public AlterTableOp visitReorderColumnsClause(ReorderColumnsClauseContext ctx) { List columnsByPos = visitIdentifierList(ctx.identifierList()); + columnsByPos.forEach(column -> requireNonEmptyColumnIdentifier(ctx.identifierList(), column)); String rollupName = ctx.fromRollup() != null ? ctx.fromRollup().rollup.getText() : null; Map properties = ctx.properties != null ? Maps.newHashMap(visitPropertyClause(ctx.properties)) @@ -6384,7 +6462,7 @@ public AlterTableOp visitRenamePartitionClause(RenamePartitionClauseContext ctx) @Override public AlterTableOp visitRenameColumnClause(RenameColumnClauseContext ctx) { - return new RenameColumnOp(ctx.name.getText(), ctx.newName.getText()); + return new RenameColumnOp(parseColumnPath(ctx.name, visitQualifiedName(ctx.name)), ctx.newName.getText()); } @Override @@ -6480,9 +6558,9 @@ public AlterTableOp visitModifyTableCommentClause(ModifyTableCommentClauseContex @Override public AlterTableOp visitModifyColumnCommentClause(ModifyColumnCommentClauseContext ctx) { - String columnName = ctx.name.getText(); - String comment = stripQuotes(ctx.STRING_LITERAL().getText()); - return new ModifyColumnCommentOp(columnName, comment); + ColumnPath columnPath = parseColumnPath(ctx.name, visitQualifiedName(ctx.name)); + String comment = SqlLiteralUtils.parseStringLiteral(ctx.STRING_LITERAL().getText()); + return new ModifyColumnCommentOp(columnPath, comment); } @Override diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilderAssistant.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilderAssistant.java index 7806201a75dbbd..63a8e9557b1144 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilderAssistant.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilderAssistant.java @@ -19,6 +19,7 @@ import org.apache.doris.nereids.trees.plans.logical.LogicalCheckPolicy; import org.apache.doris.nereids.trees.plans.logical.LogicalPlan; +import org.apache.doris.nereids.util.SqlLiteralUtils; import com.google.common.collect.ImmutableSet; @@ -42,45 +43,7 @@ private LogicalPlanBuilderAssistant() { * EscapeBackSlash such \n, \t */ public static String escapeBackSlash(String str) { - StringBuilder sb = new StringBuilder(); - int strLen = str.length(); - for (int i = 0; i < strLen; ++i) { - char c = str.charAt(i); - if (c == '\\' && (i + 1) < strLen) { - switch (str.charAt(i + 1)) { - case 'n': - sb.append('\n'); - break; - case 't': - sb.append('\t'); - break; - case 'r': - sb.append('\r'); - break; - case 'b': - sb.append('\b'); - break; - case '0': - sb.append('\0'); // Ascii null - break; - case 'Z': // ^Z must be escaped on Win32 - sb.append('\032'); - break; - case '_': - case '%': - sb.append('\\'); // remember prefix for wildcard - sb.append(str.charAt(i + 1)); - break; - default: - sb.append(str.charAt(i + 1)); - break; - } - i++; - } else { - sb.append(c); - } - } - return sb.toString(); + return SqlLiteralUtils.unescapeBackSlash(str); } /** diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/NereidsParser.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/NereidsParser.java index 37c7f9351908f7..9c71956d38f0fd 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/NereidsParser.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/NereidsParser.java @@ -61,6 +61,7 @@ import java.util.BitSet; import java.util.Iterator; import java.util.List; +import java.util.Locale; import java.util.Map; import java.util.Optional; import java.util.Set; @@ -305,13 +306,14 @@ public List> parseMultiple(String sql, } public Expression parseExpression(String expression) { - if (isSimpleIdentifier(expression)) { + if (isValidUnquotedIdentifier(expression)) { return new UnboundSlot(expression); } return parse(expression, DorisParser::expressionWithEof); } - private static boolean isSimpleIdentifier(String expression) { + /** Return whether the text can be emitted as an unquoted Nereids identifier. */ + public static boolean isValidUnquotedIdentifier(String expression) { if (expression == null || expression.isEmpty()) { return false; } @@ -328,7 +330,7 @@ private static boolean isSimpleIdentifier(String expression) { if (!hasLetter) { return false; } - String upperCase = expression.toUpperCase(); + String upperCase = expression.toUpperCase(Locale.ROOT); return (NON_RESERVED_KEYWORDS.contains(upperCase) || !LITERAL_TOKENS.containsKey(upperCase)); } diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/AlterTableCommand.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/AlterTableCommand.java index 88e684f088383c..9b7eda21119eef 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/AlterTableCommand.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/AlterTableCommand.java @@ -18,6 +18,7 @@ package org.apache.doris.nereids.trees.plans.commands; import org.apache.doris.analysis.AlterTableClause; +import org.apache.doris.analysis.ColumnPath; import org.apache.doris.analysis.ColumnPosition; import org.apache.doris.analysis.StmtType; import org.apache.doris.catalog.AggregateType; @@ -36,6 +37,7 @@ import org.apache.doris.common.UserException; import org.apache.doris.common.util.InternalDatabaseUtil; import org.apache.doris.common.util.PropertyAnalyzer; +import org.apache.doris.datasource.iceberg.IcebergExternalTable; import org.apache.doris.info.TableNameInfo; import org.apache.doris.mysql.privilege.PrivPredicate; import org.apache.doris.nereids.trees.plans.PlanType; @@ -53,6 +55,7 @@ import org.apache.doris.nereids.trees.plans.commands.info.DropRollupOp; import org.apache.doris.nereids.trees.plans.commands.info.DropTagOp; import org.apache.doris.nereids.trees.plans.commands.info.EnableFeatureOp; +import org.apache.doris.nereids.trees.plans.commands.info.ModifyColumnCommentOp; import org.apache.doris.nereids.trees.plans.commands.info.ModifyColumnOp; import org.apache.doris.nereids.trees.plans.commands.info.ModifyEngineOp; import org.apache.doris.nereids.trees.plans.commands.info.ModifyTablePropertiesOp; @@ -102,6 +105,13 @@ public List getOps() { return alterTableClauses; } + /** + * Return the path-aware operations before branch-4.1 translates them to legacy clauses. + */ + public List getNereidsOps() { + return ops; + } + /** * validate */ @@ -122,10 +132,6 @@ private void validate(ConnectContext ctx) throws UserException { if (ops == null || ops.isEmpty()) { ErrorReport.reportAnalysisException(ErrorCode.ERR_NO_ALTER_OPERATION); } - for (AlterTableOp op : ops) { - op.setTableName(tbl); - op.validate(ctx); - } String ctlName = tbl.getCtl(); String dbName = tbl.getDb(); String tableName = tbl.getTbl(); @@ -136,6 +142,11 @@ private void validate(ConnectContext ctx) throws UserException { if (tableIf.isTemporary()) { throw new AnalysisException("Do not support alter temporary table[" + tableName + "]"); } + checkColumnOperationsSupported(tableIf, ops); + for (AlterTableOp op : ops) { + op.setTableName(tbl); + op.validate(ctx); + } if (tableIf instanceof OlapTable) { rewriteAlterOpForOlapTable(ctx, (OlapTable) tableIf); } else { @@ -143,6 +154,140 @@ private void validate(ConnectContext ctx) throws UserException { } } + static void checkColumnOperationsSupported(TableIf table, List alterTableOps) + throws AnalysisException { + if (table instanceof IcebergExternalTable) { + checkIcebergCompoundColumnOperations(alterTableOps); + for (AlterTableOp alterTableOp : alterTableOps) { + ColumnDefinition columnDefinition = getColumnDefinition(alterTableOp); + ColumnPath nestedColumnPath = getNestedColumnPath(alterTableOp); + // Keep this before AddColumnOp.validate(), whose generic NOT NULL check would mask + // the Iceberg nested-field invariant. Metadata validation retains the same guard for other callers. + if (alterTableOp instanceof AddColumnOp && columnDefinition != null && nestedColumnPath != null + && !columnDefinition.isNullable()) { + throw new AnalysisException("New nested field '" + nestedColumnPath.getFullPath() + + "' must be nullable"); + } + if (!isIcebergColumnSchemaOperation(alterTableOp)) { + continue; + } + if (getRollupName(alterTableOp) != null) { + throw new AnalysisException("Rollup is not supported for Iceberg column operations"); + } + Map properties = alterTableOp.getProperties(); + if (properties != null && !properties.isEmpty()) { + throw new AnalysisException("PROPERTIES are not supported for Iceberg column operations"); + } + checkIcebergColumnDefinition(alterTableOp, columnDefinition); + if (alterTableOp instanceof AddColumnsOp) { + for (ColumnDefinition definition : ((AddColumnsOp) alterTableOp).getColumnDefinitions()) { + checkIcebergColumnDefinition(alterTableOp, definition); + } + } + } + return; + } + for (AlterTableOp alterTableOp : alterTableOps) { + ColumnPath columnPath = getNestedColumnPath(alterTableOp); + if (columnPath != null) { + throw new AnalysisException("Nested column path is only supported for Iceberg tables: " + + columnPath.getFullPath()); + } + } + } + + private static void checkIcebergCompoundColumnOperations(List alterTableOps) + throws AnalysisException { + if (alterTableOps.size() <= 1) { + return; + } + for (AlterTableOp alterTableOp : alterTableOps) { + if (isIcebergColumnSchemaOperation(alterTableOp)) { + throw new AnalysisException("Multiple Iceberg ALTER clauses are not supported when a statement " + + "contains a column operation"); + } + } + } + + private static ColumnPath getNestedColumnPath(AlterTableOp alterTableOp) { + ColumnPath columnPath = null; + if (alterTableOp instanceof AddColumnOp) { + columnPath = ((AddColumnOp) alterTableOp).getColumnPath(); + } else if (alterTableOp instanceof DropColumnOp) { + columnPath = ((DropColumnOp) alterTableOp).getColumnPath(); + } else if (alterTableOp instanceof RenameColumnOp) { + columnPath = ((RenameColumnOp) alterTableOp).getColumnPath(); + } else if (alterTableOp instanceof ModifyColumnOp) { + columnPath = ((ModifyColumnOp) alterTableOp).getColumnPath(); + } else if (alterTableOp instanceof ModifyColumnCommentOp) { + columnPath = ((ModifyColumnCommentOp) alterTableOp).getColumnPath(); + } + return columnPath != null && columnPath.isNested() ? columnPath : null; + } + + private static ColumnDefinition getColumnDefinition(AlterTableOp alterTableOp) { + if (alterTableOp instanceof AddColumnOp) { + return ((AddColumnOp) alterTableOp).getColumnDef(); + } + if (alterTableOp instanceof ModifyColumnOp) { + return ((ModifyColumnOp) alterTableOp).getColumnDef(); + } + return null; + } + + private static boolean isIcebergColumnSchemaOperation(AlterTableOp alterTableOp) { + return alterTableOp instanceof AddColumnOp + || alterTableOp instanceof AddColumnsOp + || alterTableOp instanceof DropColumnOp + || alterTableOp instanceof RenameColumnOp + || alterTableOp instanceof ModifyColumnOp + || alterTableOp instanceof ModifyColumnCommentOp + || alterTableOp instanceof ReorderColumnsOp; + } + + private static void checkIcebergColumnDefinition(AlterTableOp alterTableOp, ColumnDefinition columnDefinition) + throws AnalysisException { + if (columnDefinition == null) { + return; + } + if (columnDefinition.isKey()) { + throw new AnalysisException("KEY is not supported for Iceberg ADD/MODIFY COLUMN"); + } + if (columnDefinition.getGeneratedColumnDesc().isPresent()) { + throw new AnalysisException("Generated columns are not supported for Iceberg ADD/MODIFY COLUMN"); + } + if (alterTableOp instanceof ModifyColumnOp + && (columnDefinition.hasDefaultValue() || columnDefinition.hasOnUpdateDefaultValue())) { + columnDefinition.validateComplexTypeDefaultValue(); + throw new AnalysisException("Modifying default values is not supported for Iceberg columns: " + + ((ModifyColumnOp) alterTableOp).getColumnPath().getFullPath()); + } + if ((alterTableOp instanceof AddColumnOp || alterTableOp instanceof AddColumnsOp) + && columnDefinition.hasOnUpdateDefaultValue()) { + throw new AnalysisException("ON UPDATE is not supported for Iceberg ADD COLUMN: " + + columnDefinition.getName()); + } + } + + private static String getRollupName(AlterTableOp alterTableOp) { + if (alterTableOp instanceof AddColumnOp) { + return ((AddColumnOp) alterTableOp).getRollupName(); + } + if (alterTableOp instanceof AddColumnsOp) { + return ((AddColumnsOp) alterTableOp).getRollupName(); + } + if (alterTableOp instanceof DropColumnOp) { + return ((DropColumnOp) alterTableOp).getRollupName(); + } + if (alterTableOp instanceof ModifyColumnOp) { + return ((ModifyColumnOp) alterTableOp).getRollupName(); + } + if (alterTableOp instanceof ReorderColumnsOp) { + return ((ReorderColumnsOp) alterTableOp).getRollupName(); + } + return null; + } + private void rewriteAlterOpForOlapTable(ConnectContext ctx, OlapTable table) throws UserException { List alterTableOps = new ArrayList<>(); for (AlterTableOp alterClause : ops) { @@ -249,6 +394,7 @@ private void checkExternalTableOperationAllow(TableIf table) throws UserExceptio || alterClause instanceof DropColumnOp || alterClause instanceof RenameColumnOp || alterClause instanceof ModifyColumnOp + || (alterClause instanceof ModifyColumnCommentOp && table instanceof IcebergExternalTable) || alterClause instanceof ReorderColumnsOp || alterClause instanceof ModifyEngineOp || alterClause instanceof ModifyTablePropertiesOp diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/AddColumnOp.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/AddColumnOp.java index 00b8fea42aeae0..7b97f78b922605 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/AddColumnOp.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/AddColumnOp.java @@ -20,6 +20,7 @@ import org.apache.doris.alter.AlterOpType; import org.apache.doris.analysis.AddColumnClause; import org.apache.doris.analysis.AlterTableClause; +import org.apache.doris.analysis.ColumnPath; import org.apache.doris.analysis.ColumnPosition; import org.apache.doris.catalog.AggregateType; import org.apache.doris.catalog.Column; @@ -46,6 +47,7 @@ */ public class AddColumnOp extends AlterTableOp { private ColumnDefinition columnDef; + private ColumnPath columnPath; // Column position private ColumnPosition colPos; // if rollupName is null, add to column to base index. @@ -58,8 +60,17 @@ public class AddColumnOp extends AlterTableOp { public AddColumnOp(ColumnDefinition columnDef, ColumnPosition colPos, String rollupName, Map properties) { + this(columnDef, ColumnPath.of(columnDef.getName()), colPos, rollupName, properties); + } + + /** + * Create add-column operation with the original nested column path. + */ + public AddColumnOp(ColumnDefinition columnDef, ColumnPath columnPath, ColumnPosition colPos, + String rollupName, Map properties) { super(AlterOpType.SCHEMA_CHANGE); this.columnDef = columnDef; + this.columnPath = columnPath; this.colPos = colPos; this.rollupName = rollupName; this.properties = properties; @@ -69,6 +80,14 @@ public Column getColumn() { return column; } + public ColumnDefinition getColumnDef() { + return columnDef; + } + + public ColumnPath getColumnPath() { + return columnPath; + } + public ColumnPosition getColPos() { return colPos; } @@ -86,13 +105,13 @@ public void validate(ConnectContext ctx) throws UserException { if (colPos != null) { colPos.analyze(); } - validateColumnDef(tableName, columnDef, colPos, rollupName); + validateColumnDef(tableName, columnDef, colPos, rollupName, columnPath.isNested()); column = columnDef.translateToCatalogStyleForSchemaChange(); } @Override public AlterTableClause translateToLegacyAlterClause() { - return new AddColumnClause(toSql(), column, colPos, rollupName, properties); + return new AddColumnClause(toSql(), columnPath, column, colPos, rollupName, properties); } @Override @@ -113,7 +132,7 @@ public boolean needChangeMTMVState() { @Override public String toSql() { StringBuilder sb = new StringBuilder(); - sb.append("ADD COLUMN ").append(columnDef.toSql()); + sb.append("ADD COLUMN ").append(columnDef.toSql(columnPath.toSql())); if (colPos != null) { sb.append(" ").append(colPos.toSql()); } @@ -134,6 +153,11 @@ public String toString() { public static void validateColumnDef(TableNameInfo tableName, ColumnDefinition columnDef, ColumnPosition colPos, String rollupName) throws UserException { + validateColumnDef(tableName, columnDef, colPos, rollupName, false); + } + + private static void validateColumnDef(TableNameInfo tableName, ColumnDefinition columnDef, ColumnPosition colPos, + String rollupName, boolean nestedColumn) throws UserException { if (columnDef == null) { throw new AnalysisException("No column definition in add column clause."); } @@ -181,7 +205,11 @@ public static void validateColumnDef(TableNameInfo tableName, ColumnDefinition c .addAll(olapTable.getBaseSchema().stream().filter(Column::isClusterKey).map(Column::getName) .collect(Collectors.toList())); } - columnDef.validate(isOlap, keysSet, clusterKeySet, isEnableMergeOnWrite, keysType); + if (nestedColumn) { + columnDef.validateNestedColumn(isOlap, keysSet, clusterKeySet, isEnableMergeOnWrite, keysType); + } else { + columnDef.validate(isOlap, keysSet, clusterKeySet, isEnableMergeOnWrite, keysType); + } if (!columnDef.isNullable() && !columnDef.hasDefaultValue()) { ErrorReport.reportAnalysisException(ErrorCode.ERR_NO_DEFAULT_FOR_FIELD, columnDef.getName()); } diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/AddColumnsOp.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/AddColumnsOp.java index f04a9338aa9667..9bdd2faac1ff74 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/AddColumnsOp.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/AddColumnsOp.java @@ -64,6 +64,10 @@ public List getColumns() { return columns; } + public List getColumnDefinitions() { + return columnDefs == null ? Collections.emptyList() : columnDefs; + } + public String getRollupName() { return rollupName; } diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/ColumnDefinition.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/ColumnDefinition.java index ee2976f29e990c..81374de189f341 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/ColumnDefinition.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/ColumnDefinition.java @@ -38,6 +38,7 @@ import org.apache.doris.nereids.types.TinyIntType; import org.apache.doris.nereids.types.VarcharType; import org.apache.doris.nereids.types.coercion.CharacterType; +import org.apache.doris.nereids.util.SqlLiteralUtils; import org.apache.doris.qe.ConnectContext; import org.apache.doris.qe.ConnectContextUtil; import org.apache.doris.qe.SessionVariable; @@ -60,9 +61,13 @@ public class ColumnDefinition { private boolean isKey; private AggregateType aggType; private boolean isNullable; + // Distinguishes an explicit NULL/NOT NULL clause from the parser's default nullability. + private final boolean nullableSpecified; private Optional defaultValue; private Optional onUpdateDefaultValue = Optional.empty(); private final String comment; + // Distinguishes an explicit COMMENT '' clause from an omitted COMMENT clause. + private final boolean commentSpecified; private final boolean isVisible; private boolean aggTypeImplicit = false; private long autoIncInitValue = -1; @@ -82,7 +87,7 @@ public ColumnDefinition(String name, DataType type, boolean isKey, AggregateType Optional onUpdateDefaultValue, String comment, Optional generatedColumnDesc) { this(name, type, isKey, aggType, nullableType, autoIncInitValue, defaultValue, onUpdateDefaultValue, - comment, true, generatedColumnDesc); + comment, comment != null && !comment.isEmpty(), true, generatedColumnDesc); } /** @@ -95,8 +100,10 @@ public ColumnDefinition(String name, DataType type, boolean isKey, AggregateType this.isKey = isKey; this.aggType = aggType; this.isNullable = isNullable; + this.nullableSpecified = true; this.defaultValue = defaultValue; this.comment = comment; + this.commentSpecified = comment != null && !comment.isEmpty(); this.isVisible = isVisible; } @@ -111,10 +118,12 @@ private ColumnDefinition(String name, DataType type, boolean isKey, AggregateTyp this.isKey = isKey; this.aggType = aggType; this.isNullable = isNullable; + this.nullableSpecified = true; this.autoIncInitValue = autoIncInitValue; this.defaultValue = defaultValue; this.onUpdateDefaultValue = onUpdateDefaultValue; this.comment = comment; + this.commentSpecified = comment != null && !comment.isEmpty(); this.isVisible = isVisible; } @@ -125,15 +134,30 @@ public ColumnDefinition(String name, DataType type, boolean isKey, AggregateType ColumnNullableType nullableType, long autoIncInitValue, Optional defaultValue, Optional onUpdateDefaultValue, String comment, boolean isVisible, Optional generatedColumnDesc) { + this(name, type, isKey, aggType, nullableType, autoIncInitValue, defaultValue, onUpdateDefaultValue, + comment, comment != null && !comment.isEmpty(), isVisible, generatedColumnDesc); + } + + /** + * constructor + */ + public ColumnDefinition(String name, DataType type, boolean isKey, AggregateType aggType, + ColumnNullableType nullableType, long autoIncInitValue, Optional defaultValue, + Optional onUpdateDefaultValue, String comment, boolean commentSpecified, + boolean isVisible, + Optional generatedColumnDesc) { this.name = name; this.type = type; this.isKey = isKey; this.aggType = aggType; this.isNullable = nullableType.getNullable(type.toCatalogDataType().getPrimitiveType()); + this.nullableSpecified = nullableType == ColumnNullableType.NULLABLE + || nullableType == ColumnNullableType.NOT_NULLABLE; this.autoIncInitValue = autoIncInitValue; this.defaultValue = defaultValue; this.onUpdateDefaultValue = onUpdateDefaultValue; this.comment = comment; + this.commentSpecified = commentSpecified; this.isVisible = isVisible; this.generatedColumnDesc = generatedColumnDesc; } @@ -182,6 +206,10 @@ public boolean hasDefaultValue() { return defaultValue.isPresent(); } + public boolean hasOnUpdateDefaultValue() { + return onUpdateDefaultValue.isPresent(); + } + public boolean isVisible() { return isVisible; } @@ -202,23 +230,40 @@ public String getComment(boolean escapeQuota) { return SqlUtils.escapeQuota(comment); } + public boolean isCommentSpecified() { + return commentSpecified; + } + /** * toSql */ public String toSql() { + return toSql("`" + name + "`", true); + } + + /** + * Convert this column definition to schema-change SQL with a caller-provided column name. + * Unlike {@link #toSql()}, this overload emits COMMENT only when it was explicitly specified. + */ + public String toSql(String columnNameSql) { + return toSql(columnNameSql, commentSpecified); + } + + private String toSql(String columnNameSql, boolean includeComment) { StringBuilder sb = new StringBuilder(); - sb.append("`").append(name).append("` "); + sb.append(columnNameSql).append(" "); sb.append(type.toSql()).append(" "); if (aggType != null && aggType != AggregateType.NONE) { sb.append(aggType.name()).append(" "); } - if (!isNullable) { - sb.append("NOT NULL "); - } else { - // should append NULL to make result can be executed right. - sb.append("NULL "); + if (nullableSpecified) { + if (!isNullable) { + sb.append("NOT NULL "); + } else { + sb.append("NULL "); + } } if (autoIncInitValue != -1) { @@ -246,7 +291,9 @@ public String toSql() { sb.append("DEFAULT ").append("NULL").append(" "); } } - sb.append("COMMENT \"").append(SqlUtils.escapeQuota(comment)).append("\""); + if (includeComment) { + sb.append("COMMENT ").append(SqlLiteralUtils.quoteStringLiteral(getComment())); + } return sb.toString(); } @@ -305,10 +352,25 @@ private void checkKeyColumnType(boolean isOlap) { */ public void validate(boolean isOlap, Set keysSet, Set clusterKeySet, boolean isEnableMergeOnWrite, KeysType keysType) { + validateInternal(isOlap, keysSet, clusterKeySet, isEnableMergeOnWrite, keysType, false); + } + + /** + * Validate a nested field whose name is scoped by its parent path rather than the Doris top-level column namespace. + */ + public void validateNestedColumn(boolean isOlap, Set keysSet, Set clusterKeySet, + boolean isEnableMergeOnWrite, KeysType keysType) { + validateInternal(isOlap, keysSet, clusterKeySet, isEnableMergeOnWrite, keysType, true); + } + + private void validateInternal(boolean isOlap, Set keysSet, Set clusterKeySet, + boolean isEnableMergeOnWrite, KeysType keysType, boolean nestedColumn) { try { // if enableAddHiddenColumn is true, can add hidden column. // So does not check if the column name starts with __DORIS_ - if (enableAddHiddenColumn) { + if (nestedColumn) { + FeNameFormat.checkColumnNameBypassSystemColumnPrefix(name); + } else if (enableAddHiddenColumn) { FeNameFormat.checkColumnNameBypassHiddenColumn(name); } else { FeNameFormat.checkColumnName(name); @@ -427,18 +489,8 @@ public void validate(boolean isOlap, Set keysSet, Set clusterKey .getValue().equals(DefaultValue.ARRAY_EMPTY_DEFAULT_VALUE.getValue())) { throw new AnalysisException("Array type column default value only support null or " + DefaultValue.ARRAY_EMPTY_DEFAULT_VALUE); - } else if (type.isMapType()) { - if (defaultValue.isPresent() && defaultValue.get() != DefaultValue.NULL_DEFAULT_VALUE) { - throw new AnalysisException("Map type column default value just support null"); - } - } else if (type.isStructType()) { - if (defaultValue.isPresent() && defaultValue.get() != DefaultValue.NULL_DEFAULT_VALUE) { - throw new AnalysisException("Struct type column default value just support null"); - } - } else if (type.isJsonType() || type.isVariantType()) { - if (defaultValue.isPresent() && defaultValue.get() != DefaultValue.NULL_DEFAULT_VALUE) { - throw new AnalysisException("Json or Variant type column default value just support null"); - } + } else { + validateComplexTypeDefaultValue(); } if (!isNullable && defaultValue.isPresent() @@ -520,6 +572,22 @@ public void validate(boolean isOlap, Set keysSet, Set clusterKey validateGeneratedColumnInfo(); } + /** + * Validate non-null defaults for complex types before connector-specific validation. + */ + public void validateComplexTypeDefaultValue() throws AnalysisException { + if (!defaultValue.isPresent() || defaultValue.get() == DefaultValue.NULL_DEFAULT_VALUE) { + return; + } + if (type.isMapType()) { + throw new AnalysisException("Map type column default value just support null"); + } else if (type.isStructType()) { + throw new AnalysisException("Struct type column default value just support null"); + } else if (type.isJsonType() || type.isVariantType()) { + throw new AnalysisException("Json or Variant type column default value just support null"); + } + } + /** * translate to catalog create table stmt */ @@ -553,6 +621,8 @@ public Column translateToCatalogStyleForSchemaChange() { generatedColumnDesc.map(desc -> ConnectContextUtil.getAffectQueryResultInPlanVariables(ConnectContext.get())) .orElse(null)); + column.setNullableSpecified(nullableSpecified); + column.setCommentSpecified(commentSpecified); column.setAggregationTypeImplicit(aggTypeImplicit); return column; } diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/DropColumnOp.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/DropColumnOp.java index 74a216d265173f..d64bcd1b72acb0 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/DropColumnOp.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/DropColumnOp.java @@ -19,6 +19,7 @@ import org.apache.doris.alter.AlterOpType; import org.apache.doris.analysis.AlterTableClause; +import org.apache.doris.analysis.ColumnPath; import org.apache.doris.analysis.DropColumnClause; import org.apache.doris.catalog.Column; import org.apache.doris.catalog.Env; @@ -44,6 +45,7 @@ */ public class DropColumnOp extends AlterTableOp { private String colName; + private ColumnPath columnPath; private String rollupName; private Map properties; @@ -52,8 +54,13 @@ public class DropColumnOp extends AlterTableOp { * DropColumnOp */ public DropColumnOp(String colName, String rollupName, Map properties) { + this(ColumnPath.of(colName), rollupName, properties); + } + + public DropColumnOp(ColumnPath columnPath, String rollupName, Map properties) { super(AlterOpType.SCHEMA_CHANGE); - this.colName = colName; + this.colName = columnPath.getLeafName(); + this.columnPath = columnPath; this.rollupName = rollupName; this.properties = properties; } @@ -62,6 +69,10 @@ public String getColName() { return colName; } + public ColumnPath getColumnPath() { + return columnPath; + } + public String getRollupName() { return rollupName; } @@ -72,7 +83,7 @@ public void validate(ConnectContext ctx) throws UserException { ErrorReport.reportAnalysisException(ErrorCode.ERR_WRONG_COLUMN_NAME, colName, FeNameFormat.getColumnNameRegex()); } - if (colName.startsWith(Column.HIDDEN_COLUMN_PREFIX)) { + if (!columnPath.isNested() && colName.startsWith(Column.HIDDEN_COLUMN_PREFIX)) { throw new AnalysisException("Do not support drop hidden column"); } @@ -150,7 +161,7 @@ public void validate(ConnectContext ctx) throws UserException { @Override public AlterTableClause translateToLegacyAlterClause() { - return new DropColumnClause(colName, rollupName, properties); + return new DropColumnClause(columnPath, rollupName, properties); } @Override @@ -171,7 +182,7 @@ public boolean needChangeMTMVState() { @Override public String toSql() { StringBuilder sb = new StringBuilder(); - sb.append("DROP COLUMN `").append(colName).append("`"); + sb.append("DROP COLUMN ").append(columnPath.toSql()); if (rollupName != null) { sb.append(" FROM `").append(rollupName).append("`"); } diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/ModifyColumnCommentOp.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/ModifyColumnCommentOp.java index 8553732f8dd5df..21c5419b30bcdf 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/ModifyColumnCommentOp.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/ModifyColumnCommentOp.java @@ -19,9 +19,11 @@ import org.apache.doris.alter.AlterOpType; import org.apache.doris.analysis.AlterTableClause; +import org.apache.doris.analysis.ColumnPath; import org.apache.doris.analysis.ModifyColumnCommentClause; import org.apache.doris.common.AnalysisException; import org.apache.doris.common.UserException; +import org.apache.doris.nereids.util.SqlLiteralUtils; import org.apache.doris.qe.ConnectContext; import com.google.common.base.Strings; @@ -33,17 +35,25 @@ * ModifyColumnCommentOp */ public class ModifyColumnCommentOp extends AlterTableOp { - private String colName; + private ColumnPath columnPath; private String comment; public ModifyColumnCommentOp(String colName, String comment) { + this(ColumnPath.of(colName), comment); + } + + public ModifyColumnCommentOp(ColumnPath columnPath, String comment) { super(AlterOpType.MODIFY_COLUMN_COMMENT); - this.colName = colName; + this.columnPath = columnPath; this.comment = Strings.nullToEmpty(comment); } public String getColName() { - return colName; + return columnPath.getFullPath(); + } + + public ColumnPath getColumnPath() { + return columnPath; } public String getComment() { @@ -57,14 +67,14 @@ public Map getProperties() { @Override public void validate(ConnectContext ctx) throws UserException { - if (Strings.isNullOrEmpty(colName)) { + if (columnPath == null || Strings.isNullOrEmpty(columnPath.getFullPath())) { throw new AnalysisException("Empty column name"); } } @Override public AlterTableClause translateToLegacyAlterClause() { - return new ModifyColumnCommentClause(colName, comment); + return new ModifyColumnCommentClause(columnPath, comment); } @Override @@ -80,8 +90,8 @@ public boolean needChangeMTMVState() { @Override public String toSql() { StringBuilder sb = new StringBuilder(); - sb.append("MODIFY COLUMN COMMENT ").append(colName); - sb.append(" '").append(comment).append("'"); + sb.append("MODIFY COLUMN ").append(columnPath.toSql()); + sb.append(" COMMENT ").append(SqlLiteralUtils.quoteStringLiteral(comment)); return sb.toString(); } diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/ModifyColumnOp.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/ModifyColumnOp.java index 7a964dcebe4014..d34708e162b047 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/ModifyColumnOp.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/ModifyColumnOp.java @@ -19,6 +19,7 @@ import org.apache.doris.alter.AlterOpType; import org.apache.doris.analysis.AlterTableClause; +import org.apache.doris.analysis.ColumnPath; import org.apache.doris.analysis.ColumnPosition; import org.apache.doris.analysis.ModifyColumnClause; import org.apache.doris.catalog.Column; @@ -45,6 +46,7 @@ */ public class ModifyColumnOp extends AlterTableOp { private ColumnDefinition columnDef; + private ColumnPath columnPath; private ColumnPosition colPos; // which rollup is to be modify, if rollup is null, modify base table. private String rollupName; @@ -56,8 +58,17 @@ public class ModifyColumnOp extends AlterTableOp { public ModifyColumnOp(ColumnDefinition columnDef, ColumnPosition colPos, String rollup, Map properties) { + this(columnDef, ColumnPath.of(columnDef.getName()), colPos, rollup, properties); + } + + /** + * Create modify-column operation with the original nested column path. + */ + public ModifyColumnOp(ColumnDefinition columnDef, ColumnPath columnPath, ColumnPosition colPos, + String rollup, Map properties) { super(AlterOpType.SCHEMA_CHANGE); this.columnDef = columnDef; + this.columnPath = columnPath; this.colPos = colPos; this.rollupName = rollup; this.properties = properties; @@ -67,6 +78,10 @@ public Column getColumn() { return column; } + public ColumnPath getColumnPath() { + return columnPath; + } + public ColumnPosition getColPos() { return colPos; } @@ -124,7 +139,11 @@ public void validate(ConnectContext ctx) throws UserException { } } } - columnDef.validate(isOlap, keysSet, clusterKeySet, isEnableMergeOnWrite, keysType); + if (columnPath.isNested()) { + columnDef.validateNestedColumn(isOlap, keysSet, clusterKeySet, isEnableMergeOnWrite, keysType); + } else { + columnDef.validate(isOlap, keysSet, clusterKeySet, isEnableMergeOnWrite, keysType); + } if (colPos != null) { colPos.analyze(); if (olapTable != null) { @@ -173,7 +192,7 @@ public ColumnDefinition getColumnDef() { @Override public AlterTableClause translateToLegacyAlterClause() { - return new ModifyColumnClause(toSql(), column, colPos, rollupName, properties); + return new ModifyColumnClause(toSql(), columnPath, column, colPos, rollupName, properties); } @Override @@ -194,7 +213,7 @@ public boolean needChangeMTMVState() { @Override public String toSql() { StringBuilder sb = new StringBuilder(); - sb.append("MODIFY COLUMN ").append(columnDef.toSql()); + sb.append("MODIFY COLUMN ").append(columnDef.toSql(columnPath.toSql())); if (colPos != null) { sb.append(" ").append(colPos); } diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/RenameColumnOp.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/RenameColumnOp.java index 6774645795f0cb..b473f860e6307f 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/RenameColumnOp.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/RenameColumnOp.java @@ -19,11 +19,13 @@ import org.apache.doris.alter.AlterOpType; import org.apache.doris.analysis.AlterTableClause; +import org.apache.doris.analysis.ColumnPath; import org.apache.doris.analysis.ColumnRenameClause; import org.apache.doris.catalog.Column; import org.apache.doris.common.AnalysisException; import org.apache.doris.common.FeNameFormat; import org.apache.doris.common.UserException; +import org.apache.doris.common.util.SqlUtils; import org.apache.doris.qe.ConnectContext; import com.google.common.base.Strings; @@ -35,11 +37,17 @@ */ public class RenameColumnOp extends AlterTableOp { private String colName; + private ColumnPath columnPath; private String newColName; public RenameColumnOp(String colName, String newColName) { + this(ColumnPath.of(colName), newColName); + } + + public RenameColumnOp(ColumnPath columnPath, String newColName) { super(AlterOpType.RENAME); - this.colName = colName; + this.colName = columnPath.getLeafName(); + this.columnPath = columnPath; this.newColName = newColName; this.needTableStable = false; } @@ -48,6 +56,10 @@ public String getColName() { return colName; } + public ColumnPath getColumnPath() { + return columnPath; + } + public String getNewColName() { return newColName; } @@ -62,16 +74,20 @@ public void validate(ConnectContext ctx) throws UserException { throw new AnalysisException("New column name is not set"); } - if (colName.startsWith(Column.HIDDEN_COLUMN_PREFIX)) { + if (!columnPath.isNested() && colName.startsWith(Column.HIDDEN_COLUMN_PREFIX)) { throw new AnalysisException("Do not support rename hidden column"); } - FeNameFormat.checkColumnName(newColName); + if (columnPath.isNested()) { + FeNameFormat.checkColumnNameBypassSystemColumnPrefix(newColName); + } else { + FeNameFormat.checkColumnName(newColName); + } } @Override public AlterTableClause translateToLegacyAlterClause() { - return new ColumnRenameClause(colName, newColName); + return new ColumnRenameClause(columnPath, newColName); } @Override @@ -91,7 +107,7 @@ public boolean needChangeMTMVState() { @Override public String toSql() { - return "RENAME COLUMN " + colName + " " + newColName; + return "RENAME COLUMN " + columnPath.toSql() + " " + SqlUtils.getIdentSql(newColName); } @Override diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/types/DataType.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/types/DataType.java index a2adc9a9fcccf8..0b50b05de219c7 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/types/DataType.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/types/DataType.java @@ -467,7 +467,8 @@ public static DataType fromCatalogType(Type type) { if (type.isStructType()) { List structFields = ((org.apache.doris.catalog.StructType) (type)).getFields().stream() .map(cf -> new StructField(cf.getName(), fromCatalogType(cf.getType()), - cf.getContainsNull(), cf.getComment() == null ? "" : cf.getComment())) + cf.getContainsNull(), cf.getComment() == null ? "" : cf.getComment(), + cf.isCommentSpecified())) .collect(ImmutableList.toImmutableList()); return new StructType(structFields); } else if (type.isMapType()) { diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/types/StructField.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/types/StructField.java index 63e94bc369bb0c..aefcf20f227ff1 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/types/StructField.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/types/StructField.java @@ -17,6 +17,9 @@ package org.apache.doris.nereids.types; +import org.apache.doris.common.util.SqlUtils; +import org.apache.doris.nereids.parser.NereidsParser; +import org.apache.doris.nereids.util.SqlLiteralUtils; import org.apache.doris.nereids.util.Utils; import java.util.Objects; @@ -32,6 +35,7 @@ public class StructField { private final DataType dataType; private final boolean nullable; private final String comment; + private final boolean commentSpecified; /** * StructField Constructor @@ -40,10 +44,16 @@ public class StructField { * @param nullable Indicates if values of this field can be `null` values */ public StructField(String name, DataType dataType, boolean nullable, String comment) { + this(name, dataType, nullable, comment, comment != null && !comment.isEmpty()); + } + + public StructField(String name, DataType dataType, boolean nullable, String comment, + boolean commentSpecified) { this.name = Objects.requireNonNull(name, "name should not be null").toLowerCase(); this.dataType = Objects.requireNonNull(dataType, "dataType should not be null"); this.nullable = nullable; this.comment = Objects.requireNonNull(comment, "comment should not be null"); + this.commentSpecified = commentSpecified; } public String getName() { @@ -62,6 +72,10 @@ public String getComment() { return comment; } + public boolean isCommentSpecified() { + return commentSpecified; + } + public StructField conversion() { if (this.dataType.equals(dataType.conversion())) { return this; @@ -70,22 +84,24 @@ public StructField conversion() { } public StructField withDataType(DataType dataType) { - return new StructField(name, dataType, nullable, comment); + return new StructField(name, dataType, nullable, comment, commentSpecified); } public StructField withDataTypeAndNullable(DataType dataType, boolean nullable) { - return new StructField(name, dataType, nullable, comment); + return new StructField(name, dataType, nullable, comment, commentSpecified); } public org.apache.doris.catalog.StructField toCatalogDataType() { return new org.apache.doris.catalog.StructField( - name, dataType.toCatalogDataType(), comment, nullable); + name, dataType.toCatalogDataType(), comment, nullable, commentSpecified); } public String toSql() { - return name + ":" + dataType.toSql() + String nameSql = NereidsParser.isValidUnquotedIdentifier(name) ? name : SqlUtils.getIdentSql(name); + return nameSql + ":" + dataType.toSql() + (nullable ? "" : " NOT NULL") - + (comment.isEmpty() ? "" : " COMMENT " + comment); + + (!commentSpecified ? "" : " COMMENT " + + SqlLiteralUtils.quoteStringLiteral(comment)); } @Override diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/util/SqlLiteralUtils.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/util/SqlLiteralUtils.java new file mode 100644 index 00000000000000..5d87bc08f5f56d --- /dev/null +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/util/SqlLiteralUtils.java @@ -0,0 +1,96 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.nereids.util; + +import org.apache.doris.qe.SqlModeHelper; + +/** + * Utilities for decoding and rendering SQL string literals. + */ +public final class SqlLiteralUtils { + + private SqlLiteralUtils() { + } + + /** + * Decode backslash escape sequences used in SQL string literals. + */ + public static String unescapeBackSlash(String value) { + StringBuilder result = new StringBuilder(); + int length = value.length(); + for (int i = 0; i < length; ++i) { + char current = value.charAt(i); + if (current == '\\' && (i + 1) < length) { + switch (value.charAt(i + 1)) { + case 'n': + result.append('\n'); + break; + case 't': + result.append('\t'); + break; + case 'r': + result.append('\r'); + break; + case 'b': + result.append('\b'); + break; + case '0': + result.append('\0'); + break; + case 'Z': + result.append('\032'); + break; + case '_': + case '%': + result.append('\\'); + result.append(value.charAt(i + 1)); + break; + default: + result.append(value.charAt(i + 1)); + break; + } + i++; + } else { + result.append(current); + } + } + return result.toString(); + } + + /** + * Decode a STRING_LITERAL token according to the current SQL mode. + */ + public static String parseStringLiteral(String text) { + String value = text.substring(1, text.length() - 1); + if (text.charAt(0) == '\'') { + value = value.replace("''", "'"); + } else { + value = value.replace("\"\"", "\""); + } + return SqlModeHelper.hasNoBackSlashEscapes() ? value : unescapeBackSlash(value); + } + + /** + * Quote a value as a STRING_LITERAL that can be parsed under the current SQL mode. + */ + public static String quoteStringLiteral(String value) { + String escaped = SqlModeHelper.hasNoBackSlashEscapes() + ? value : value.replace("\\", "\\\\"); + return "\"" + escaped.replace("\"", "\"\"") + "\""; + } +} diff --git a/fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java b/fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java index df5ac1ec75b9d5..fedf7a126c02cf 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java +++ b/fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java @@ -171,6 +171,7 @@ public class SessionVariable implements Serializable, Writable { public static final String ENABLE_DISTINCT_STREAMING_AGG_FORCE_PASSTHROUGH = "enable_distinct_streaming_agg_force_passthrough"; public static final String ENABLE_BROADCAST_JOIN_FORCE_PASSTHROUGH = "enable_broadcast_join_force_passthrough"; + public static final String ENABLE_LOCAL_EXCHANGE_BEFORE_AGG = "enable_local_exchange_before_agg"; public static final String DISABLE_COLOCATE_PLAN = "disable_colocate_plan"; public static final String COLOCATE_MAX_PARALLEL_NUM = "colocate_max_parallel_num"; public static final String ENABLE_BUCKET_SHUFFLE_JOIN = "enable_bucket_shuffle_join"; @@ -1373,6 +1374,9 @@ public void checkQuerySlotCount(String slotCnt) { @VariableMgr.VarAttr(name = ENABLE_STREAMING_AGG_HASH_JOIN_FORCE_PASSTHROUGH, fuzzy = true) public boolean enableStreamingAggHashJoinForcePassthrough = true; + @VariableMgr.VarAttr(name = ENABLE_LOCAL_EXCHANGE_BEFORE_AGG, fuzzy = true) + public boolean enableLocalExchangeBeforeAgg = true; + @VariableMgr.VarAttr(name = ENABLE_DISTINCT_STREAMING_AGG_FORCE_PASSTHROUGH, fuzzy = true) public boolean enableDistinctStreamingAggForcePassthrough = true; @@ -3693,6 +3697,7 @@ public void initFuzzyModeVariables() { this.enableFileScannerV2 = random.nextBoolean(); this.disableStreamPreaggregations = random.nextBoolean(); this.enableStreamingAggHashJoinForcePassthrough = random.nextBoolean(); + this.enableLocalExchangeBeforeAgg = random.nextBoolean(); this.enableDistinctStreamingAggForcePassthrough = random.nextBoolean(); this.enableBroadcastJoinForcePassthrough = random.nextBoolean(); this.enableShareHashTableForBroadcastJoin = random.nextBoolean(); @@ -5500,6 +5505,7 @@ public TQueryOptions toThrift() { tResult.setDisableStreamPreaggregations(disableStreamPreaggregations); tResult.setEnableDistinctStreamingAggregation(enableDistinctStreamingAggregation); tResult.setEnableStreamingAggHashJoinForcePassthrough(enableStreamingAggHashJoinForcePassthrough); + tResult.setEnableLocalExchangeBeforeAgg(enableLocalExchangeBeforeAgg); tResult.setEnableDistinctStreamingAggForcePassthrough(enableDistinctStreamingAggForcePassthrough); tResult.setEnableBroadcastJoinForcePassthrough(enableBroadcastJoinForcePassthrough); tResult.setPartitionTopnMaxPartitions(partitionTopNMaxPartitions); diff --git a/fe/fe-core/src/test/java/org/apache/doris/common/util/PrintableMapTest.java b/fe/fe-core/src/test/java/org/apache/doris/common/util/PrintableMapTest.java index 0514cfedb9abd9..2a40528eb1a8af 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/common/util/PrintableMapTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/common/util/PrintableMapTest.java @@ -43,6 +43,8 @@ public void testSensitiveKeysContainAliyunDLFProperties() { Assertions.assertTrue(PrintableMap.SENSITIVE_KEY.contains("bos_secret_accesskey")); Assertions.assertTrue(PrintableMap.SENSITIVE_KEY.contains("jdbc.password")); Assertions.assertTrue(PrintableMap.SENSITIVE_KEY.contains("elasticsearch.password")); + Assertions.assertTrue(PrintableMap.SENSITIVE_KEY.contains("iceberg.rest.oauth2.credential")); + Assertions.assertTrue(PrintableMap.SENSITIVE_KEY.contains("iceberg.rest.oauth2.token")); // Verify cloud storage related sensitive keys (these are constants added in static initialization block) Assertions.assertTrue(PrintableMap.SENSITIVE_KEY.contains("s3.secret_key")); @@ -155,6 +157,8 @@ public void testHidePasswordWithSensitiveKeys() { testMap.put("password", "secret_password"); testMap.put("dlf.secret_key", "dlf_secret_value"); testMap.put("s3.secret_key", "s3_secret_value"); + testMap.put("iceberg.rest.oauth2.credential", "oauth_credential"); + testMap.put("iceberg.rest.oauth2.token", "oauth_token"); testMap.put("kerberos_keytab_content", "kerberos_content"); PrintableMap printableMap = new PrintableMap<>(testMap, "=", false, false, true); @@ -164,6 +168,10 @@ public void testHidePasswordWithSensitiveKeys() { Assertions.assertTrue(result.contains("password = " + PrintableMap.PASSWORD_MASK)); Assertions.assertTrue(result.contains("dlf.secret_key = " + PrintableMap.PASSWORD_MASK)); Assertions.assertTrue(result.contains("s3.secret_key = " + PrintableMap.PASSWORD_MASK)); + Assertions.assertTrue(result.contains( + "iceberg.rest.oauth2.credential = " + PrintableMap.PASSWORD_MASK)); + Assertions.assertTrue(result.contains( + "iceberg.rest.oauth2.token = " + PrintableMap.PASSWORD_MASK)); Assertions.assertTrue(result.contains("kerberos_keytab_content = " + PrintableMap.PASSWORD_MASK)); } diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/iceberg/IcebergMetadataOpsValidationTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/iceberg/IcebergMetadataOpsValidationTest.java index 937167c34c351d..d0236a0bdb1412 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/datasource/iceberg/IcebergMetadataOpsValidationTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/iceberg/IcebergMetadataOpsValidationTest.java @@ -17,6 +17,8 @@ package org.apache.doris.datasource.iceberg; +import org.apache.doris.analysis.ColumnPath; +import org.apache.doris.analysis.ColumnPosition; import org.apache.doris.catalog.ArrayType; import org.apache.doris.catalog.Column; import org.apache.doris.catalog.MapType; @@ -27,34 +29,57 @@ import org.apache.doris.common.UserException; import org.apache.doris.common.security.authentication.ExecutionAuthenticator; import org.apache.doris.datasource.ExternalCatalog; +import org.apache.doris.datasource.ExternalTable; +import org.apache.hadoop.conf.Configuration; +import org.apache.iceberg.CatalogProperties; +import org.apache.iceberg.Schema; +import org.apache.iceberg.Table; +import org.apache.iceberg.UpdateSchema; import org.apache.iceberg.catalog.Catalog; +import org.apache.iceberg.catalog.Namespace; import org.apache.iceberg.catalog.SupportsNamespaces; +import org.apache.iceberg.catalog.TableIdentifier; +import org.apache.iceberg.exceptions.CommitFailedException; +import org.apache.iceberg.hadoop.HadoopCatalog; import org.apache.iceberg.types.Types; import org.apache.iceberg.types.Types.NestedField; import org.junit.Assert; import org.junit.Before; +import org.junit.Rule; import org.junit.Test; +import org.junit.rules.TemporaryFolder; +import org.mockito.MockedStatic; import org.mockito.Mockito; import java.lang.reflect.InvocationTargetException; import java.lang.reflect.Method; +import java.util.Arrays; import java.util.Collections; +import java.util.HashMap; +import java.util.Map; +import java.util.Optional; +import java.util.concurrent.atomic.AtomicBoolean; public class IcebergMetadataOpsValidationTest { private IcebergMetadataOps ops; + private ExternalCatalog dorisCatalog; private Method validateForModifyColumnMethod; private Method validateForModifyComplexColumnMethod; + @Rule + public TemporaryFolder temporaryFolder = new TemporaryFolder(); + @Before public void setUp() throws Exception { - ExternalCatalog dorisCatalog = Mockito.mock(ExternalCatalog.class); + dorisCatalog = Mockito.mock(ExternalCatalog.class); Catalog icebergCatalog = Mockito.mock(Catalog.class, Mockito.withSettings().extraInterfaces(SupportsNamespaces.class)); Mockito.when(dorisCatalog.getExecutionAuthenticator()).thenReturn(new ExecutionAuthenticator() { }); Mockito.when(dorisCatalog.getProperties()).thenReturn(Collections.emptyMap()); + Mockito.doReturn(Optional.empty()).when(dorisCatalog).getDbForReplay(Mockito.anyString()); ops = new IcebergMetadataOps(dorisCatalog, icebergCatalog); validateForModifyColumnMethod = IcebergMetadataOps.class.getDeclaredMethod( @@ -88,6 +113,15 @@ public void testValidateForModifyColumnSuccess() throws Throwable { invokeValidateForModifyColumn(column, currentCol); } + @Test + public void testValidateForModifyColumnRejectsComplexToPrimitive() { + Column column = new Column("struct_col", Type.INT, true); + NestedField currentCol = Types.NestedField.required(1, "struct_col", Types.StructType.of( + Types.NestedField.required(2, "value", Types.IntegerType.get()))); + assertUserException(() -> invokeValidateForModifyColumn(column, currentCol), + "Modify column type from complex to primitive is not supported: struct_col"); + } + @Test public void testValidateForModifyComplexColumnRejectsPrimitiveType() { Column column = new Column("arr_i", Type.INT, true); @@ -182,6 +216,1193 @@ public void testValidateForModifyComplexColumnSuccess() throws Throwable { invokeValidateForModifyComplexColumn(column, currentCol); } + @Test + public void testRejectUnsupportedIcebergTargetTypesBeforeUpdateSchema() { + Schema schema = requiredNestedSchema(); + ExternalTable dorisTable = Mockito.mock(ExternalTable.class); + Table icebergTable = Mockito.mock(Table.class); + Mockito.when(icebergTable.schema()).thenReturn(schema); + + StructType unsupportedStruct = new StructType(new StructField("value", Type.LARGEINT)); + ArrayType unsupportedArray = ArrayType.create(Type.LARGEINT, true); + MapType unsupportedMap = new MapType(Type.STRING, Type.LARGEINT); + + try (MockedStatic mockedIcebergUtils = + Mockito.mockStatic(IcebergUtils.class, Mockito.CALLS_REAL_METHODS)) { + mockedIcebergUtils.when(() -> IcebergUtils.getIcebergTable(dorisTable)).thenReturn(icebergTable); + + assertUserException(() -> ops.addColumn(dorisTable, ColumnPath.fromDotName("info.new_field"), + new Column("new_field", Type.LARGEINT, true), null, 1L), + "is not supported for Iceberg column new_field"); + assertUserException(() -> ops.modifyColumn(dorisTable, ColumnPath.fromDotName("info.metric"), + new Column("metric", Type.LARGEINT, true), null, 1L), + "is not supported for Iceberg column info.metric"); + assertUserException(() -> ops.modifyColumn(dorisTable, ColumnPath.fromDotName("info.child"), + new Column("child", unsupportedStruct, false), null, 1L), + "is not supported for Iceberg column info.child.value"); + assertUserException(() -> ops.modifyColumn(dorisTable, ColumnPath.fromDotName("info.events"), + new Column("events", unsupportedArray, false), null, 1L), + "is not supported for Iceberg column info.events.element"); + assertUserException(() -> ops.modifyColumn(dorisTable, ColumnPath.fromDotName("info.attrs"), + new Column("attrs", unsupportedMap, false), null, 1L), + "is not supported for Iceberg column info.attrs.value"); + } + + Mockito.verify(icebergTable, Mockito.never()).updateSchema(); + } + + @Test + public void testComplexModifyPreservesRequiredNestedFields() throws Throwable { + Schema schema = requiredNestedSchema(); + ExternalTable dorisTable = Mockito.mock(ExternalTable.class); + Table icebergTable = Mockito.mock(Table.class); + UpdateSchema updateSchema = Mockito.mock(UpdateSchema.class); + Mockito.when(dorisTable.getRemoteDbName()).thenReturn("db"); + Mockito.when(icebergTable.schema()).thenReturn(schema); + Mockito.when(icebergTable.updateSchema()).thenReturn(updateSchema); + + try (MockedStatic mockedIcebergUtils = + Mockito.mockStatic(IcebergUtils.class, Mockito.CALLS_REAL_METHODS)) { + mockedIcebergUtils.when(() -> IcebergUtils.getIcebergTable(dorisTable)).thenReturn(icebergTable); + + ops.modifyColumn(dorisTable, ColumnPath.fromDotName("info.child"), + new Column("child", new StructType(new StructField("value", Type.BIGINT)), true), null, 1L); + ops.modifyColumn(dorisTable, ColumnPath.fromDotName("info.events"), + new Column("events", ArrayType.create(Type.BIGINT, true), true), null, 1L); + ops.modifyColumn(dorisTable, ColumnPath.fromDotName("info.attrs"), + new Column("attrs", new MapType(Type.STRING, Type.BIGINT), true), null, 1L); + } + + Mockito.verify(updateSchema).updateColumn("info.child.value", Types.LongType.get(), null); + Mockito.verify(updateSchema).updateColumn("info.events.element", Types.LongType.get(), null); + Mockito.verify(updateSchema).updateColumn("info.attrs.value", Types.LongType.get(), null); + Mockito.verify(updateSchema, Mockito.never()).makeColumnOptional(Mockito.anyString()); + Mockito.verify(updateSchema, Mockito.times(3)).commit(); + } + + @Test + public void testComplexModifyPersistsDecodedStructMemberComment() throws Throwable { + Schema schema = new Schema(Types.NestedField.optional(1, "info", Types.StructType.of( + Types.NestedField.optional(2, "payload", Types.StructType.of( + Types.NestedField.optional(3, "name", Types.StringType.get(), "old comment")))))); + ExternalTable dorisTable = Mockito.mock(ExternalTable.class); + Table icebergTable = Mockito.mock(Table.class); + UpdateSchema updateSchema = Mockito.mock(UpdateSchema.class); + Mockito.when(dorisTable.getRemoteDbName()).thenReturn("db"); + Mockito.when(icebergTable.schema()).thenReturn(schema); + Mockito.when(icebergTable.updateSchema()).thenReturn(updateSchema); + + String decodedComment = "owner's \"field\" C:\\tmp\\"; + Column column = new Column("payload", new StructType( + new StructField("name", Type.STRING, decodedComment, true)), true); + + try (MockedStatic mockedIcebergUtils = + Mockito.mockStatic(IcebergUtils.class, Mockito.CALLS_REAL_METHODS)) { + mockedIcebergUtils.when(() -> IcebergUtils.getIcebergTable(dorisTable)).thenReturn(icebergTable); + + ops.modifyColumn(dorisTable, ColumnPath.fromDotName("info.payload"), column, null, 1L); + } + + Mockito.verify(updateSchema).updateColumnDoc("info.payload.name", decodedComment); + Mockito.verify(updateSchema).commit(); + } + + @Test + public void testPrimitiveModifyPreservesOmittedCommentAndClearsExplicitEmptyComment() throws Throwable { + Schema schema = new Schema( + Types.NestedField.optional(1, "info", Types.StructType.of( + Types.NestedField.optional(2, "metric", Types.IntegerType.get(), "metric doc"), + Types.NestedField.optional(3, "clear_me", Types.StringType.get(), "clear doc"))), + Types.NestedField.optional(4, "top_metric", Types.IntegerType.get(), "top metric doc")); + ExternalTable dorisTable = Mockito.mock(ExternalTable.class); + Table icebergTable = Mockito.mock(Table.class); + UpdateSchema updateSchema = Mockito.mock(UpdateSchema.class); + Mockito.when(dorisTable.getRemoteDbName()).thenReturn("db"); + Mockito.when(icebergTable.schema()).thenReturn(schema); + Mockito.when(icebergTable.updateSchema()).thenReturn(updateSchema); + + Column clearComment = new Column("clear_me", Type.STRING, true, ""); + clearComment.setCommentSpecified(true); + + try (MockedStatic mockedIcebergUtils = + Mockito.mockStatic(IcebergUtils.class, Mockito.CALLS_REAL_METHODS)) { + mockedIcebergUtils.when(() -> IcebergUtils.getIcebergTable(dorisTable)).thenReturn(icebergTable); + + ops.modifyColumn(dorisTable, ColumnPath.fromDotName("info.metric"), + new Column("metric", Type.BIGINT, true), null, 1L); + ops.modifyColumn(dorisTable, ColumnPath.of("top_metric"), + new Column("top_metric", Type.BIGINT, true), null, 1L); + ops.modifyColumn(dorisTable, ColumnPath.fromDotName("info.clear_me"), + clearComment, null, 1L); + } + + Mockito.verify(updateSchema).updateColumn("info.metric", Types.LongType.get(), "metric doc"); + Mockito.verify(updateSchema).updateColumn("top_metric", Types.LongType.get(), "top metric doc"); + Mockito.verify(updateSchema).updateColumnDoc("info.clear_me", ""); + Mockito.verify(updateSchema, Mockito.times(3)).commit(); + } + + @Test + public void testFullStructModifyPreservesOmittedChildComments() throws Throwable { + Schema schema = new Schema(Types.NestedField.optional(1, "info", Types.StructType.of( + Types.NestedField.optional(2, "payload", Types.StructType.of( + Types.NestedField.optional(3, "metric", Types.IntegerType.get(), "metric doc"), + Types.NestedField.optional(4, "clear_me", Types.StringType.get(), "clear doc"), + Types.NestedField.optional(5, "details", Types.StructType.of( + Types.NestedField.optional( + 6, "count", Types.IntegerType.get(), "count doc")), "details doc")), + "payload doc")))); + ExternalTable dorisTable = Mockito.mock(ExternalTable.class); + Table icebergTable = Mockito.mock(Table.class); + UpdateSchema updateSchema = Mockito.mock(UpdateSchema.class); + Mockito.when(dorisTable.getRemoteDbName()).thenReturn("db"); + Mockito.when(icebergTable.schema()).thenReturn(schema); + Mockito.when(icebergTable.updateSchema()).thenReturn(updateSchema); + + StructType detailsType = new StructType( + new StructField("count", Type.BIGINT, "", true, false)); + StructType payloadType = new StructType( + new StructField("metric", Type.BIGINT, "", true, false), + new StructField("clear_me", Type.STRING, "", true, true), + new StructField("details", detailsType, "", true, false)); + + try (MockedStatic mockedIcebergUtils = + Mockito.mockStatic(IcebergUtils.class, Mockito.CALLS_REAL_METHODS)) { + mockedIcebergUtils.when(() -> IcebergUtils.getIcebergTable(dorisTable)).thenReturn(icebergTable); + + ops.modifyColumn(dorisTable, ColumnPath.fromDotName("info.payload"), + new Column("payload", payloadType, true), null, 1L); + } + + Mockito.verify(updateSchema).updateColumn( + "info.payload.metric", Types.LongType.get(), "metric doc"); + Mockito.verify(updateSchema).updateColumnDoc("info.payload.clear_me", ""); + Mockito.verify(updateSchema).updateColumn( + "info.payload.details.count", Types.LongType.get(), "count doc"); + Mockito.verify(updateSchema, Mockito.never()).updateColumnDoc( + Mockito.eq("info.payload"), Mockito.nullable(String.class)); + Mockito.verify(updateSchema, Mockito.never()).updateColumnDoc( + Mockito.eq("info.payload.details"), Mockito.nullable(String.class)); + Mockito.verify(updateSchema).commit(); + } + + @Test + public void testPrimitiveModifyPreservesRequiredNestedField() throws Throwable { + Schema schema = requiredNestedSchema(); + ExternalTable dorisTable = Mockito.mock(ExternalTable.class); + Table icebergTable = Mockito.mock(Table.class); + UpdateSchema updateSchema = Mockito.mock(UpdateSchema.class); + Mockito.when(dorisTable.getRemoteDbName()).thenReturn("db"); + Mockito.when(icebergTable.schema()).thenReturn(schema); + Mockito.when(icebergTable.updateSchema()).thenReturn(updateSchema); + + try (MockedStatic mockedIcebergUtils = + Mockito.mockStatic(IcebergUtils.class, Mockito.CALLS_REAL_METHODS)) { + mockedIcebergUtils.when(() -> IcebergUtils.getIcebergTable(dorisTable)).thenReturn(icebergTable); + + ops.modifyColumn(dorisTable, ColumnPath.fromDotName("info.metric"), + new Column("metric", Type.BIGINT, true), null, 1L); + } + + Mockito.verify(updateSchema).updateColumn("info.metric", Types.LongType.get(), null); + Mockito.verify(updateSchema, Mockito.never()).makeColumnOptional(Mockito.anyString()); + Mockito.verify(updateSchema).commit(); + } + + @Test + public void testTopLevelModifyPreservesRequiredMixedCaseFields() throws Throwable { + Schema schema = new Schema( + Types.NestedField.required(1, "Id", Types.IntegerType.get()), + Types.NestedField.required(2, "Payload", Types.StructType.of( + Types.NestedField.required(3, "Value", Types.IntegerType.get())))); + ExternalTable dorisTable = Mockito.mock(ExternalTable.class); + Table icebergTable = Mockito.mock(Table.class); + UpdateSchema updateSchema = Mockito.mock(UpdateSchema.class); + Mockito.when(dorisTable.getRemoteDbName()).thenReturn("db"); + Mockito.when(icebergTable.schema()).thenReturn(schema); + Mockito.when(icebergTable.updateSchema()).thenReturn(updateSchema); + + try (MockedStatic mockedIcebergUtils = + Mockito.mockStatic(IcebergUtils.class, Mockito.CALLS_REAL_METHODS)) { + mockedIcebergUtils.when(() -> IcebergUtils.getIcebergTable(dorisTable)).thenReturn(icebergTable); + + ops.modifyColumn(dorisTable, ColumnPath.of("id"), + new Column("id", Type.BIGINT, true), null, 1L); + ops.modifyColumn(dorisTable, ColumnPath.of("payload"), new Column("payload", + new StructType(new StructField("Value", Type.BIGINT)), true), null, 1L); + } + + Mockito.verify(updateSchema).updateColumn("Id", Types.LongType.get(), null); + Mockito.verify(updateSchema).updateColumn("Payload.Value", Types.LongType.get(), null); + Mockito.verify(updateSchema, Mockito.never()).makeColumnOptional(Mockito.anyString()); + Mockito.verify(updateSchema, Mockito.times(2)).commit(); + } + + @Test + public void testTopLevelModifyDoesNotResolveQuotedComponentAsNestedPath() { + Schema schema = new Schema( + Types.NestedField.optional(1, "a", Types.StructType.of( + Types.NestedField.optional(2, "b", Types.IntegerType.get()))), + Types.NestedField.optional(3, "b", Types.IntegerType.get())); + ExternalTable dorisTable = Mockito.mock(ExternalTable.class); + Table icebergTable = Mockito.mock(Table.class); + Mockito.when(icebergTable.schema()).thenReturn(schema); + + try (MockedStatic mockedIcebergUtils = + Mockito.mockStatic(IcebergUtils.class, Mockito.CALLS_REAL_METHODS)) { + mockedIcebergUtils.when(() -> IcebergUtils.getIcebergTable(dorisTable)).thenReturn(icebergTable); + + assertUserException(() -> ops.modifyColumn(dorisTable, ColumnPath.of("a.b"), + new Column("a.b", Type.BIGINT, true), null, 1L), + "Column a.b does not exist"); + } + + Mockito.verify(icebergTable, Mockito.never()).updateSchema(); + } + + @Test + public void testTopLevelModifyPreservesDottedTopLevelName() throws Throwable { + Schema schema = new Schema(Types.NestedField.optional(1, "a.b", Types.IntegerType.get())); + ExternalTable dorisTable = Mockito.mock(ExternalTable.class); + Table icebergTable = Mockito.mock(Table.class); + UpdateSchema updateSchema = Mockito.mock(UpdateSchema.class); + Mockito.when(dorisTable.getRemoteDbName()).thenReturn("db"); + Mockito.when(icebergTable.schema()).thenReturn(schema); + Mockito.when(icebergTable.updateSchema()).thenReturn(updateSchema); + + try (MockedStatic mockedIcebergUtils = + Mockito.mockStatic(IcebergUtils.class, Mockito.CALLS_REAL_METHODS)) { + mockedIcebergUtils.when(() -> IcebergUtils.getIcebergTable(dorisTable)).thenReturn(icebergTable); + + ops.modifyColumn(dorisTable, ColumnPath.of("a.b"), + new Column("a.b", Type.BIGINT, true), null, 1L); + } + + Mockito.verify(updateSchema).updateColumn("a.b", Types.LongType.get(), null); + Mockito.verify(updateSchema).commit(); + } + + @Test + public void testPrimitiveModifyPreservesActualTypeWhenMappingDisabled() throws Throwable { + Schema schema = mappedPrimitiveSchema(); + ExternalTable dorisTable = Mockito.mock(ExternalTable.class); + Table icebergTable = Mockito.mock(Table.class); + UpdateSchema updateSchema = Mockito.mock(UpdateSchema.class); + Mockito.when(dorisTable.getRemoteDbName()).thenReturn("db"); + Mockito.when(icebergTable.schema()).thenReturn(schema); + Mockito.when(icebergTable.updateSchema()).thenReturn(updateSchema); + Mockito.when(dorisCatalog.getEnableMappingVarbinary()).thenReturn(false); + Mockito.when(dorisCatalog.getEnableMappingTimestampTz()).thenReturn(false); + + Column topUuid = new Column("top_uuid", Type.STRING, true); + topUuid.setNullableSpecified(true); + Column nestedUuid = new Column("uuid_value", Type.STRING, true); + nestedUuid.setNullableSpecified(true); + Column nestedTimestamp = new Column( + "tz_value", ScalarType.createDatetimeV2Type(6), true); + + try (MockedStatic mockedIcebergUtils = + Mockito.mockStatic(IcebergUtils.class, Mockito.CALLS_REAL_METHODS)) { + mockedIcebergUtils.when(() -> IcebergUtils.getIcebergTable(dorisTable)).thenReturn(icebergTable); + + ops.modifyColumn(dorisTable, ColumnPath.of("top_uuid"), topUuid, ColumnPosition.FIRST, 1L); + ops.modifyColumn(dorisTable, ColumnPath.fromDotName("info.uuid_value"), nestedUuid, + new ColumnPosition("other"), 1L); + ops.modifyColumn(dorisTable, ColumnPath.fromDotName("info.tz_value"), + nestedTimestamp, null, 1L); + } + + Mockito.verify(updateSchema, Mockito.never()).updateColumnDoc( + Mockito.anyString(), Mockito.nullable(String.class)); + Mockito.verify(updateSchema).makeColumnOptional("top_uuid"); + Mockito.verify(updateSchema).makeColumnOptional("info.uuid_value"); + Mockito.verify(updateSchema).moveFirst("top_uuid"); + Mockito.verify(updateSchema).moveAfter("info.uuid_value", "info.other"); + Mockito.verify(updateSchema, Mockito.never()).updateColumn( + Mockito.anyString(), Mockito.any(org.apache.iceberg.types.Type.PrimitiveType.class), + Mockito.nullable(String.class)); + Mockito.verify(updateSchema, Mockito.times(3)).commit(); + } + + @Test + public void testPrimitiveModifyPreservesActualTypeWhenMappingEnabled() throws Throwable { + Schema schema = mappedPrimitiveSchema(); + ExternalTable dorisTable = Mockito.mock(ExternalTable.class); + Table icebergTable = Mockito.mock(Table.class); + UpdateSchema updateSchema = Mockito.mock(UpdateSchema.class); + Mockito.when(dorisTable.getRemoteDbName()).thenReturn("db"); + Mockito.when(icebergTable.schema()).thenReturn(schema); + Mockito.when(icebergTable.updateSchema()).thenReturn(updateSchema); + Mockito.when(dorisCatalog.getEnableMappingVarbinary()).thenReturn(true); + Mockito.when(dorisCatalog.getEnableMappingTimestampTz()).thenReturn(true); + + try (MockedStatic mockedIcebergUtils = + Mockito.mockStatic(IcebergUtils.class, Mockito.CALLS_REAL_METHODS)) { + mockedIcebergUtils.when(() -> IcebergUtils.getIcebergTable(dorisTable)).thenReturn(icebergTable); + + ops.modifyColumn(dorisTable, ColumnPath.of("top_uuid"), + new Column("top_uuid", ScalarType.createVarbinaryType(16), true), null, 1L); + ops.modifyColumn(dorisTable, ColumnPath.fromDotName("info.tz_value"), + new Column("tz_value", ScalarType.createTimeStampTzType(6), true), null, 1L); + } + + Mockito.verify(updateSchema, Mockito.never()).updateColumnDoc( + Mockito.anyString(), Mockito.nullable(String.class)); + Mockito.verify(updateSchema, Mockito.never()).updateColumn( + Mockito.anyString(), Mockito.any(org.apache.iceberg.types.Type.PrimitiveType.class), + Mockito.nullable(String.class)); + Mockito.verify(updateSchema, Mockito.times(2)).commit(); + } + + @Test + public void testComplexModifyIgnoresUnchangedMappedChildren() throws Throwable { + Schema schema = mappedComplexSchema(); + ExternalTable dorisTable = Mockito.mock(ExternalTable.class); + Table icebergTable = Mockito.mock(Table.class); + UpdateSchema updateSchema = Mockito.mock(UpdateSchema.class); + Mockito.when(dorisTable.getRemoteDbName()).thenReturn("db"); + Mockito.when(icebergTable.schema()).thenReturn(schema); + Mockito.when(icebergTable.updateSchema()).thenReturn(updateSchema); + Mockito.when(dorisCatalog.getEnableMappingVarbinary()).thenReturn(true); + Mockito.when(dorisCatalog.getEnableMappingTimestampTz()).thenReturn(true); + + try (MockedStatic mockedIcebergUtils = + Mockito.mockStatic(IcebergUtils.class, Mockito.CALLS_REAL_METHODS)) { + mockedIcebergUtils.when(() -> IcebergUtils.getIcebergTable(dorisTable)).thenReturn(icebergTable); + + ops.modifyColumn(dorisTable, ColumnPath.fromDotName("outer.payload"), + new Column("payload", mappedPayloadDorisType(Type.BIGINT, 8, + ScalarType.createVarbinaryType(4)), true), null, 1L); + } + + Mockito.verify(updateSchema).updateColumn( + "outer.payload.metric", Types.LongType.get(), null); + Mockito.verify(updateSchema, Mockito.times(1)).updateColumn( + Mockito.anyString(), Mockito.any(org.apache.iceberg.types.Type.PrimitiveType.class), + Mockito.nullable(String.class)); + Mockito.verify(updateSchema, Mockito.never()).updateColumnDoc( + Mockito.anyString(), Mockito.nullable(String.class)); + Mockito.verify(updateSchema).commit(); + } + + @Test + public void testComplexModifyRejectsChangedUnsupportedMappedChildrenBeforeUpdateSchema() { + Schema schema = mappedComplexSchema(); + ExternalTable dorisTable = Mockito.mock(ExternalTable.class); + Table icebergTable = Mockito.mock(Table.class); + Mockito.when(icebergTable.schema()).thenReturn(schema); + Mockito.when(dorisCatalog.getEnableMappingVarbinary()).thenReturn(true); + Mockito.when(dorisCatalog.getEnableMappingTimestampTz()).thenReturn(true); + + try (MockedStatic mockedIcebergUtils = + Mockito.mockStatic(IcebergUtils.class, Mockito.CALLS_REAL_METHODS)) { + mockedIcebergUtils.when(() -> IcebergUtils.getIcebergTable(dorisTable)).thenReturn(icebergTable); + + assertUserException(() -> ops.modifyColumn(dorisTable, ColumnPath.fromDotName("outer.payload"), + new Column("payload", mappedPayloadDorisType(Type.LARGEINT, 8, + ScalarType.createVarbinaryType(4)), true), null, 1L), + "Type largeint is not supported for Iceberg column outer.payload.metric"); + assertUserException(() -> ops.modifyColumn(dorisTable, ColumnPath.fromDotName("outer.payload"), + new Column("payload", mappedPayloadDorisType(Type.INT, 16, + ScalarType.createVarbinaryType(4)), true), null, 1L), + "Type varbinary(16) is not supported for Iceberg column outer.payload.fixed_value"); + assertUserException(() -> ops.modifyColumn(dorisTable, ColumnPath.fromDotName("outer.payload"), + new Column("payload", mappedPayloadDorisType(Type.INT, 8, + ScalarType.createVarbinaryType(8)), true), null, 1L), + "Cannot change MAP key type from varbinary(4) to varbinary(8)"); + } + + Mockito.verify(icebergTable, Mockito.never()).updateSchema(); + } + + @Test + public void testLegacyModifyColumnTreatsNullabilityAsExplicit() throws Throwable { + Schema schema = new Schema(Types.NestedField.required(1, "id", Types.IntegerType.get())); + ExternalTable dorisTable = Mockito.mock(ExternalTable.class); + Table icebergTable = Mockito.mock(Table.class); + UpdateSchema updateSchema = Mockito.mock(UpdateSchema.class); + Mockito.when(dorisTable.getRemoteDbName()).thenReturn("db"); + Mockito.when(icebergTable.schema()).thenReturn(schema); + Mockito.when(icebergTable.updateSchema()).thenReturn(updateSchema); + + try (MockedStatic mockedIcebergUtils = + Mockito.mockStatic(IcebergUtils.class, Mockito.CALLS_REAL_METHODS)) { + mockedIcebergUtils.when(() -> IcebergUtils.getIcebergTable(dorisTable)).thenReturn(icebergTable); + + // Iceberg schema columns are represented as keys in Doris, so the legacy API must not + // interpret isKey as an explicit KEY clause. + ops.modifyColumn(dorisTable, + new Column("id", Type.BIGINT, true, null, true, null, ""), null, 1L); + } + + Mockito.verify(updateSchema).updateColumn("id", Types.LongType.get(), ""); + Mockito.verify(updateSchema).makeColumnOptional("id"); + Mockito.verify(updateSchema).commit(); + } + + @Test + public void testLegacyComplexModifyDoesNotInferRecursiveNullableChanges() throws Throwable { + Schema schema = requiredNestedSchema(); + ExternalTable dorisTable = Mockito.mock(ExternalTable.class); + Table icebergTable = Mockito.mock(Table.class); + UpdateSchema updateSchema = Mockito.mock(UpdateSchema.class); + Mockito.when(dorisTable.getRemoteDbName()).thenReturn("db"); + Mockito.when(icebergTable.schema()).thenReturn(schema); + Mockito.when(icebergTable.updateSchema()).thenReturn(updateSchema); + + Column column = new Column("info", new StructType( + new StructField("metric", Type.INT), + new StructField("child", new StructType(new StructField("value", Type.INT))), + new StructField("events", ArrayType.create(Type.INT, true)), + new StructField("attrs", new MapType(Type.STRING, Type.INT))), true); + + try (MockedStatic mockedIcebergUtils = + Mockito.mockStatic(IcebergUtils.class, Mockito.CALLS_REAL_METHODS)) { + mockedIcebergUtils.when(() -> IcebergUtils.getIcebergTable(dorisTable)).thenReturn(icebergTable); + + ops.modifyColumn(dorisTable, column, null, 1L); + } + + Mockito.verify(updateSchema).makeColumnOptional("info"); + Mockito.verify(updateSchema, Mockito.never()).makeColumnOptional("info.metric"); + Mockito.verify(updateSchema, Mockito.never()).makeColumnOptional("info.child.value"); + Mockito.verify(updateSchema, Mockito.never()).makeColumnOptional("info.events.element"); + Mockito.verify(updateSchema, Mockito.never()).makeColumnOptional("info.attrs.value"); + Mockito.verify(updateSchema).commit(); + } + + @Test + public void testExplicitNullableModifyMakesRequiredFieldsOptional() throws Throwable { + Schema schema = requiredNestedSchema(); + ExternalTable dorisTable = Mockito.mock(ExternalTable.class); + Table icebergTable = Mockito.mock(Table.class); + UpdateSchema updateSchema = Mockito.mock(UpdateSchema.class); + Mockito.when(dorisTable.getRemoteDbName()).thenReturn("db"); + Mockito.when(icebergTable.schema()).thenReturn(schema); + Mockito.when(icebergTable.updateSchema()).thenReturn(updateSchema); + + Column topLevelColumn = new Column("info", new StructType( + new StructField("metric", Type.BIGINT), + new StructField("child", new StructType(new StructField("value", Type.INT))), + new StructField("events", ArrayType.create(Type.INT, true)), + new StructField("attrs", new MapType(Type.STRING, Type.INT))), true); + topLevelColumn.setNullableSpecified(true); + Column nestedColumn = new Column("metric", Type.BIGINT, true); + nestedColumn.setNullableSpecified(true); + + try (MockedStatic mockedIcebergUtils = + Mockito.mockStatic(IcebergUtils.class, Mockito.CALLS_REAL_METHODS)) { + mockedIcebergUtils.when(() -> IcebergUtils.getIcebergTable(dorisTable)).thenReturn(icebergTable); + + ops.modifyColumn(dorisTable, ColumnPath.of("info"), topLevelColumn, null, 1L); + ops.modifyColumn(dorisTable, ColumnPath.fromDotName("info.metric"), nestedColumn, null, 1L); + } + + Mockito.verify(updateSchema).makeColumnOptional("info"); + Mockito.verify(updateSchema).makeColumnOptional("info.metric"); + Mockito.verify(updateSchema, Mockito.times(2)).commit(); + } + + @Test + public void testSchemaCommitConflictDoesNotPartiallyApplyComplexModify() throws Exception { + String dbName = "db"; + String tableName = "conflict_table"; + TableIdentifier tableIdentifier = TableIdentifier.of(dbName, tableName); + Map properties = new HashMap<>(); + properties.put(CatalogProperties.WAREHOUSE_LOCATION, + temporaryFolder.newFolder("iceberg_warehouse").toURI().toString()); + + HadoopCatalog icebergCatalog = new HadoopCatalog(); + icebergCatalog.setConf(new Configuration()); + icebergCatalog.initialize("conflict_catalog", properties); + icebergCatalog.createNamespace(Namespace.of(dbName)); + Table staleTable = icebergCatalog.createTable(tableIdentifier, new Schema( + Types.NestedField.optional(1, "info", Types.StructType.of( + Types.NestedField.optional(2, "a", Types.IntegerType.get()), + Types.NestedField.optional(3, "b", Types.IntegerType.get()))))); + Table concurrentTable = icebergCatalog.loadTable(tableIdentifier); + + ExternalCatalog conflictDorisCatalog = Mockito.mock(ExternalCatalog.class); + AtomicBoolean conflictInjected = new AtomicBoolean(false); + Mockito.when(conflictDorisCatalog.getExecutionAuthenticator()).thenReturn(new ExecutionAuthenticator() { + @Override + public void execute(Runnable task) { + Assert.assertTrue("schema commit should execute only once", conflictInjected.compareAndSet(false, true)); + concurrentTable.updateSchema().renameColumn("info.a", "concurrent_a").commit(); + task.run(); + } + }); + Mockito.when(conflictDorisCatalog.getProperties()).thenReturn(Collections.emptyMap()); + Mockito.doReturn(Optional.empty()).when(conflictDorisCatalog).getDbForReplay(Mockito.anyString()); + IcebergMetadataOps conflictOps = new IcebergMetadataOps(conflictDorisCatalog, icebergCatalog); + + ExternalTable dorisTable = Mockito.mock(ExternalTable.class); + Mockito.when(dorisTable.getRemoteDbName()).thenReturn(dbName); + Mockito.when(dorisTable.getRemoteName()).thenReturn(tableName); + StructType promotedInfo = new StructType( + new StructField("a", Type.BIGINT), + new StructField("b", Type.BIGINT)); + + try (MockedStatic mockedIcebergUtils = + Mockito.mockStatic(IcebergUtils.class, Mockito.CALLS_REAL_METHODS)) { + mockedIcebergUtils.when(() -> IcebergUtils.getIcebergTable(dorisTable)).thenReturn(staleTable); + + try { + conflictOps.modifyColumn(dorisTable, ColumnPath.of("info"), + new Column("info", promotedInfo, true), null, 1L); + Assert.fail("expected schema commit conflict"); + } catch (UserException e) { + Assert.assertTrue(e.getMessage().contains("Failed to modify column: info")); + Assert.assertTrue("expected Iceberg commit conflict but got " + e.getCause(), + e.getCause() instanceof CommitFailedException); + } + } + + Schema committedSchema = icebergCatalog.loadTable(tableIdentifier).schema(); + Assert.assertNull(committedSchema.findField("info.a")); + Assert.assertEquals(Types.IntegerType.get(), committedSchema.findType("info.concurrent_a")); + Assert.assertEquals(Types.IntegerType.get(), committedSchema.findType("info.b")); + Mockito.verify(conflictDorisCatalog, Mockito.never()).getDbForReplay(Mockito.anyString()); + + icebergCatalog.dropTable(tableIdentifier); + icebergCatalog.dropNamespace(Namespace.of(dbName)); + icebergCatalog.close(); + } + + @Test + public void testRenamePreservesNestedIdentifierFieldPaths() throws Exception { + String dbName = "db"; + String tableName = "identifier_table"; + TableIdentifier tableIdentifier = TableIdentifier.of(dbName, tableName); + Map properties = new HashMap<>(); + properties.put(CatalogProperties.WAREHOUSE_LOCATION, + temporaryFolder.newFolder("identifier_warehouse").toURI().toString()); + + HadoopCatalog icebergCatalog = new HadoopCatalog(); + icebergCatalog.setConf(new Configuration()); + icebergCatalog.initialize("identifier_catalog", properties); + icebergCatalog.createNamespace(Namespace.of(dbName)); + Schema schema = new Schema(Arrays.asList( + Types.NestedField.required(1, "root", Types.StructType.of( + Types.NestedField.required(2, "child", Types.StructType.of( + Types.NestedField.required(3, "id", Types.IntegerType.get()), + Types.NestedField.optional(4, "value", Types.StringType.get())))))), + Collections.singleton(3)); + Table icebergTable = icebergCatalog.createTable(tableIdentifier, schema); + + ExternalTable dorisTable = Mockito.mock(ExternalTable.class); + Mockito.when(dorisTable.getRemoteDbName()).thenReturn(dbName); + Mockito.when(dorisTable.getRemoteName()).thenReturn(tableName); + + try (MockedStatic mockedIcebergUtils = + Mockito.mockStatic(IcebergUtils.class, Mockito.CALLS_REAL_METHODS)) { + mockedIcebergUtils.when(() -> IcebergUtils.getIcebergTable(dorisTable)).thenReturn(icebergTable); + + ops.renameColumn(dorisTable, ColumnPath.fromDotName("root.child.id"), "renamed_id", 1L); + icebergTable.refresh(); + Assert.assertEquals(Collections.singleton("root.child.renamed_id"), + icebergTable.schema().identifierFieldNames()); + + ops.renameColumn(dorisTable, ColumnPath.fromDotName("root.child"), "renamed_child", 1L); + icebergTable.refresh(); + Assert.assertEquals(Collections.singleton("root.renamed_child.renamed_id"), + icebergTable.schema().identifierFieldNames()); + + ops.renameColumn(dorisTable, "root", "renamed_root", 1L); + icebergTable.refresh(); + Assert.assertEquals(Collections.singleton("renamed_root.renamed_child.renamed_id"), + icebergTable.schema().identifierFieldNames()); + Assert.assertEquals(3, icebergTable.schema().findField( + "renamed_root.renamed_child.renamed_id").fieldId()); + } + + icebergCatalog.dropTable(tableIdentifier); + icebergCatalog.dropNamespace(Namespace.of(dbName)); + icebergCatalog.close(); + } + + @Test + public void testRenameDoesNotRewriteDottedIdentifierSibling() throws Exception { + String dbName = "db"; + String tableName = "dotted_identifier_table"; + TableIdentifier tableIdentifier = TableIdentifier.of(dbName, tableName); + Map properties = new HashMap<>(); + properties.put(CatalogProperties.WAREHOUSE_LOCATION, + temporaryFolder.newFolder("dotted_identifier_warehouse").toURI().toString()); + + HadoopCatalog icebergCatalog = new HadoopCatalog(); + icebergCatalog.setConf(new Configuration()); + icebergCatalog.initialize("dotted_identifier_catalog", properties); + icebergCatalog.createNamespace(Namespace.of(dbName)); + Schema schema = new Schema(Arrays.asList( + Types.NestedField.required(1, "a", Types.IntegerType.get()), + Types.NestedField.required(2, "a.b", Types.IntegerType.get())), + Collections.singleton(2)); + Table icebergTable = icebergCatalog.createTable(tableIdentifier, schema); + + ExternalTable dorisTable = Mockito.mock(ExternalTable.class); + Mockito.when(dorisTable.getRemoteDbName()).thenReturn(dbName); + Mockito.when(dorisTable.getRemoteName()).thenReturn(tableName); + + try (MockedStatic mockedIcebergUtils = + Mockito.mockStatic(IcebergUtils.class, Mockito.CALLS_REAL_METHODS)) { + mockedIcebergUtils.when(() -> IcebergUtils.getIcebergTable(dorisTable)).thenReturn(icebergTable); + + ops.renameColumn(dorisTable, "a", "renamed", 1L); + icebergTable.refresh(); + Assert.assertNotNull(icebergTable.schema().findField("renamed")); + Assert.assertEquals(Collections.singleton("a.b"), icebergTable.schema().identifierFieldNames()); + Assert.assertEquals(2, icebergTable.schema().findField("a.b").fieldId()); + } + + icebergCatalog.dropTable(tableIdentifier); + icebergCatalog.dropNamespace(Namespace.of(dbName)); + icebergCatalog.close(); + } + + @Test + public void testNestedColumnOperationsRejectDefaultMetadata() { + Schema schema = new Schema(Types.NestedField.optional(1, "s", Types.StructType.of( + Types.NestedField.optional(2, "existing", Types.IntegerType.get())))); + ExternalTable dorisTable = Mockito.mock(ExternalTable.class); + Table icebergTable = Mockito.mock(Table.class); + Mockito.when(icebergTable.schema()).thenReturn(schema); + + Column nestedAddDefaultColumn = new Column("new_col", Type.BIGINT, false, null, true, "7", ""); + Column nestedDefaultColumn = new Column("existing", Type.BIGINT, false, null, true, "7", ""); + Column nestedOnUpdateColumn = Mockito.spy(new Column("existing", Type.BIGINT, true)); + Mockito.doReturn(true).when(nestedOnUpdateColumn).hasOnUpdateDefaultValue(); + + try (MockedStatic mockedIcebergUtils = + Mockito.mockStatic(IcebergUtils.class, Mockito.CALLS_REAL_METHODS)) { + mockedIcebergUtils.when(() -> IcebergUtils.getIcebergTable(dorisTable)).thenReturn(icebergTable); + + assertUserException(() -> ops.addColumn(dorisTable, ColumnPath.fromDotName("s.new_col"), + nestedAddDefaultColumn, null, 1L), + "DEFAULT and ON UPDATE are not supported for Iceberg nested ADD COLUMN: s.new_col"); + assertUserException(() -> ops.modifyColumn(dorisTable, ColumnPath.fromDotName("s.existing"), + nestedDefaultColumn, null, 1L), + "Modifying default values is not supported for Iceberg columns: s.existing"); + assertUserException(() -> ops.modifyColumn(dorisTable, ColumnPath.fromDotName("s.existing"), + nestedOnUpdateColumn, null, 1L), + "Modifying default values is not supported for Iceberg columns: s.existing"); + } + + Mockito.verify(icebergTable, Mockito.never()).updateSchema(); + } + + @Test + public void testTopLevelColumnOperationsRejectUnsupportedDefaultMetadata() { + Schema schema = new Schema(Types.NestedField.optional(1, "id", Types.IntegerType.get())); + ExternalTable dorisTable = Mockito.mock(ExternalTable.class); + Table icebergTable = Mockito.mock(Table.class); + Mockito.when(icebergTable.schema()).thenReturn(schema); + + Column defaultColumn = new Column("id", Type.BIGINT, false, null, true, "7", ""); + Column onUpdateColumn = Mockito.spy(new Column("id", Type.BIGINT, true)); + Column onUpdateAddColumn = Mockito.spy(new Column("new_col", Type.BIGINT, true)); + Mockito.doReturn(true).when(onUpdateColumn).hasOnUpdateDefaultValue(); + Mockito.doReturn(true).when(onUpdateAddColumn).hasOnUpdateDefaultValue(); + + try (MockedStatic mockedIcebergUtils = + Mockito.mockStatic(IcebergUtils.class, Mockito.CALLS_REAL_METHODS)) { + mockedIcebergUtils.when(() -> IcebergUtils.getIcebergTable(dorisTable)).thenReturn(icebergTable); + + assertUserException(() -> ops.modifyColumn(dorisTable, defaultColumn, null, 1L), + "Modifying default values is not supported for Iceberg columns: id"); + assertUserException(() -> ops.modifyColumn( + dorisTable, ColumnPath.of("id"), onUpdateColumn, null, 1L), + "Modifying default values is not supported for Iceberg columns: id"); + assertUserException(() -> ops.addColumn(dorisTable, onUpdateAddColumn, null, 1L), + "ON UPDATE is not supported for Iceberg ADD COLUMN: new_col"); + assertUserException(() -> ops.addColumns( + dorisTable, Collections.singletonList(onUpdateAddColumn), 1L), + "ON UPDATE is not supported for Iceberg ADD COLUMN: new_col"); + } + + Mockito.verify(icebergTable, Mockito.never()).updateSchema(); + } + + @Test + public void testUnsupportedPrimitiveModifyFailsBeforeUpdateSchema() { + Schema schema = new Schema( + Types.NestedField.required(1, "top_long", Types.LongType.get()), + Types.NestedField.required(2, "info", Types.StructType.of( + Types.NestedField.required(3, "metric", Types.LongType.get()), + Types.NestedField.required(4, "child", Types.StructType.of( + Types.NestedField.required(5, "value", Types.IntegerType.get())))))); + ExternalTable dorisTable = Mockito.mock(ExternalTable.class); + Table icebergTable = Mockito.mock(Table.class); + Mockito.when(icebergTable.schema()).thenReturn(schema); + + try (MockedStatic mockedIcebergUtils = + Mockito.mockStatic(IcebergUtils.class, Mockito.CALLS_REAL_METHODS)) { + mockedIcebergUtils.when(() -> IcebergUtils.getIcebergTable(dorisTable)).thenReturn(icebergTable); + + assertUserException(() -> ops.modifyColumn( + dorisTable, ColumnPath.of("info"), new Column("info", Type.INT, true), null, 1L), + "Modify column type from complex to primitive is not supported: info"); + assertUserException(() -> ops.modifyColumn(dorisTable, ColumnPath.fromDotName("info.child"), + new Column("child", Type.INT, true), null, 1L), + "Modify column type from complex to primitive is not supported: info.child"); + assertUserException(() -> ops.modifyColumn(dorisTable, ColumnPath.of("top_long"), + new Column("top_long", Type.INT, true), null, 1L), + "Cannot change column type: top_long: long -> int"); + assertUserException(() -> ops.modifyColumn(dorisTable, ColumnPath.fromDotName("info.metric"), + new Column("metric", Type.INT, true), null, 1L), + "Cannot change column type: info.metric: long -> int"); + } + + Mockito.verify(icebergTable, Mockito.never()).updateSchema(); + } + + @Test + public void testRejectKeyAndGeneratedMetadataBeforeUpdateSchema() { + Schema schema = new Schema(Types.NestedField.optional(1, "id", Types.IntegerType.get())); + ExternalTable dorisTable = Mockito.mock(ExternalTable.class); + Table icebergTable = Mockito.mock(Table.class); + Mockito.when(icebergTable.schema()).thenReturn(schema); + + Column keyColumn = new Column("id", Type.BIGINT, true, null, true, null, ""); + Column generatedColumn = Mockito.mock(Column.class); + Mockito.when(generatedColumn.getName()).thenReturn("id"); + Mockito.when(generatedColumn.isGeneratedColumn()).thenReturn(true); + + try (MockedStatic mockedIcebergUtils = + Mockito.mockStatic(IcebergUtils.class, Mockito.CALLS_REAL_METHODS)) { + mockedIcebergUtils.when(() -> IcebergUtils.getIcebergTable(dorisTable)).thenReturn(icebergTable); + + assertUserException(() -> ops.addColumn(dorisTable, keyColumn, null, 1L), + "KEY is not supported for Iceberg ADD/MODIFY COLUMN"); + assertUserException(() -> ops.addColumns(dorisTable, Collections.singletonList(keyColumn), 1L), + "KEY is not supported for Iceberg ADD/MODIFY COLUMN"); + assertUserException(() -> ops.modifyColumn( + dorisTable, ColumnPath.of("id"), keyColumn, null, 1L), + "KEY is not supported for Iceberg ADD/MODIFY COLUMN"); + assertUserException(() -> ops.addColumns(dorisTable, + Collections.singletonList(generatedColumn), 1L), + "Generated columns are not supported for Iceberg ADD/MODIFY COLUMN"); + assertUserException(() -> ops.modifyColumn( + dorisTable, ColumnPath.of("id"), generatedColumn, null, 1L), + "Generated columns are not supported for Iceberg ADD/MODIFY COLUMN"); + } + + Mockito.verify(icebergTable, Mockito.never()).updateSchema(); + } + + @Test + public void testModifyComplexColumnRejectsCaseInsensitiveStructFieldAdditions() { + Schema schema = mixedCaseNestedSchema(); + ExternalTable dorisTable = Mockito.mock(ExternalTable.class); + Table icebergTable = Mockito.mock(Table.class); + UpdateSchema updateSchema = Mockito.mock(UpdateSchema.class); + Mockito.when(icebergTable.schema()).thenReturn(schema); + Mockito.when(icebergTable.updateSchema()).thenReturn(updateSchema); + + StructType infoType = new StructType( + new StructField("Metric", Type.INT), + new StructField("Label", Type.STRING), + new StructField("metric", Type.INT)); + ArrayType eventsType = ArrayType.create(new StructType( + new StructField("Score", Type.INT), + new StructField("score", Type.INT)), true); + MapType attrsType = new MapType(Type.STRING, new StructType( + new StructField("Code", Type.INT), + new StructField("code", Type.INT))); + StructType duplicateNewFieldsType = new StructType( + new StructField("Metric", Type.INT), + new StructField("Label", Type.STRING), + new StructField("Extra", Type.INT), + new StructField("EXTRA", Type.STRING)); + + try (MockedStatic mockedIcebergUtils = + Mockito.mockStatic(IcebergUtils.class, Mockito.CALLS_REAL_METHODS)) { + mockedIcebergUtils.when(() -> IcebergUtils.getIcebergTable(dorisTable)).thenReturn(icebergTable); + + assertUserException(() -> ops.modifyColumn( + dorisTable, new Column("info", infoType, true), null, 1L), + "Added struct field 'metric' conflicts with existing field"); + assertUserException(() -> ops.modifyColumn( + dorisTable, new Column("events", eventsType, true), null, 1L), + "Added struct field 'score' conflicts with existing field"); + assertUserException(() -> ops.modifyColumn( + dorisTable, new Column("attrs", attrsType, true), null, 1L), + "Added struct field 'code' conflicts with existing field"); + assertUserException(() -> ops.modifyColumn( + dorisTable, new Column("info", duplicateNewFieldsType, true), null, 1L), + "Added struct field 'extra' conflicts with existing field"); + } + + Mockito.verifyNoInteractions(updateSchema); + } + + @Test + public void testResolveNestedColumnPathSupportsStructArrayElementAndMapValue() throws Throwable { + Schema schema = nestedSchema(); + Assert.assertTrue(ops.resolveNestedColumnPath(schema, ColumnPath.fromDotName("s"), "add").isStructType()); + Assert.assertTrue(ops.resolveNestedColumnPath(schema, ColumnPath.fromDotName("arr.element"), "add") + .isStructType()); + Assert.assertTrue(ops.resolveNestedColumnPath(schema, ColumnPath.fromDotName("m.value"), "add") + .isStructType()); + } + + @Test + public void testResolveNestedColumnPathUsesCaseInsensitiveCanonicalIcebergPath() throws Throwable { + Schema schema = mixedCaseNestedSchema(); + Assert.assertEquals("Info.Metric", + ops.getCanonicalColumnPath(schema, ColumnPath.fromDotName("info.metric"), "modify")); + Assert.assertEquals("Events.element.Score", + ops.getCanonicalColumnPath(schema, ColumnPath.fromDotName("events.element.score"), "modify")); + Assert.assertEquals("Attrs.value.Code", + ops.getCanonicalColumnPath(schema, ColumnPath.fromDotName("attrs.value.code"), "modify")); + Assert.assertEquals("Info.Label", + ops.getPositionReferencePath(schema, ColumnPath.fromDotName("Info.NewField"), + new ColumnPosition("label"), "add")); + } + + @Test + public void testValidateNoCaseInsensitiveSiblingCollisionRejectsAddAndRenameTargets() { + Types.StructType parentType = mixedCaseNestedSchema().findField("Info").type().asStructType(); + ColumnPath parentPath = ColumnPath.fromDotName("Info"); + assertUserException(() -> ops.validateNoCaseInsensitiveSiblingCollision( + parentType, parentPath, "metric", null, "add"), + "Cannot add nested column 'Info.metric': conflicts with existing Iceberg field 'Info.Metric'"); + assertUserException(() -> ops.validateNoCaseInsensitiveSiblingCollision( + parentType, parentPath, "metric", parentType.field("Label"), "rename"), + "Cannot rename nested column 'Info.metric': conflicts with existing Iceberg field 'Info.Metric'"); + } + + @Test + public void testValidateNoCaseInsensitiveSiblingCollisionAllowsCaseOnlyRename() throws Throwable { + Types.StructType parentType = mixedCaseNestedSchema().findField("Info").type().asStructType(); + ops.validateNoCaseInsensitiveSiblingCollision(parentType, ColumnPath.fromDotName("Info"), + "metric", parentType.field("Metric"), "rename"); + } + + @Test + public void testTopLevelCaseInsensitiveCollisionsAndCaseOnlyRename() throws Throwable { + Schema schema = new Schema( + Types.NestedField.optional(1, "Id", Types.IntegerType.get()), + Types.NestedField.optional(2, "Label", Types.StringType.get())); + ExternalTable dorisTable = Mockito.mock(ExternalTable.class); + Table icebergTable = Mockito.mock(Table.class); + UpdateSchema updateSchema = Mockito.mock(UpdateSchema.class); + Mockito.when(dorisTable.getRemoteDbName()).thenReturn("db"); + Mockito.when(icebergTable.schema()).thenReturn(schema); + Mockito.when(icebergTable.updateSchema()).thenReturn(updateSchema); + + try (MockedStatic mockedIcebergUtils = + Mockito.mockStatic(IcebergUtils.class, Mockito.CALLS_REAL_METHODS)) { + mockedIcebergUtils.when(() -> IcebergUtils.getIcebergTable(dorisTable)).thenReturn(icebergTable); + + assertUserException(() -> ops.addColumn( + dorisTable, new Column("id", Type.STRING, true), null, 1L), + "Cannot add column 'id': conflicts with existing Iceberg field 'Id'"); + assertUserException(() -> ops.addColumns(dorisTable, + Collections.singletonList(new Column("id", Type.STRING, true)), 1L), + "Cannot add column 'id': conflicts with existing Iceberg field 'Id'"); + assertUserException(() -> ops.addColumns(dorisTable, Arrays.asList( + new Column("new_field", Type.STRING, true), + new Column("NEW_FIELD", Type.STRING, true)), 1L), + "conflicts with another requested column (case-insensitive)"); + assertUserException(() -> ops.renameColumn(dorisTable, "label", "id", 1L), + "Cannot rename column 'id': conflicts with existing Iceberg field 'Id'"); + + ops.renameColumn(dorisTable, "id", "id", 1L); + } + + Mockito.verify(updateSchema).renameColumn("Id", "id"); + Mockito.verify(updateSchema).commit(); + } + + @Test + public void testReorderColumnsUsesCanonicalIcebergNames() throws Throwable { + Schema schema = new Schema( + Types.NestedField.optional(1, "Id", Types.IntegerType.get()), + Types.NestedField.optional(2, "Label", Types.StringType.get())); + ExternalTable dorisTable = Mockito.mock(ExternalTable.class); + Table icebergTable = Mockito.mock(Table.class); + UpdateSchema updateSchema = Mockito.mock(UpdateSchema.class); + Mockito.when(dorisTable.getRemoteDbName()).thenReturn("db"); + Mockito.when(icebergTable.schema()).thenReturn(schema); + Mockito.when(icebergTable.updateSchema()).thenReturn(updateSchema); + + try (MockedStatic mockedIcebergUtils = + Mockito.mockStatic(IcebergUtils.class, Mockito.CALLS_REAL_METHODS)) { + mockedIcebergUtils.when(() -> IcebergUtils.getIcebergTable(dorisTable)).thenReturn(icebergTable); + + ops.reorderColumns(dorisTable, Arrays.asList("label", "id"), 1L); + } + + Mockito.verify(updateSchema).moveFirst("Label"); + Mockito.verify(updateSchema).moveAfter("Id", "Label"); + Mockito.verify(updateSchema).commit(); + } + + @Test + public void testModifyColumnSupportsDirectArrayElementAndMapValue() throws Throwable { + Schema schema = primitiveContainerSchema(); + ExternalTable dorisTable = Mockito.mock(ExternalTable.class); + Table icebergTable = Mockito.mock(Table.class); + UpdateSchema updateSchema = Mockito.mock(UpdateSchema.class); + Mockito.when(dorisTable.getRemoteDbName()).thenReturn("db"); + Mockito.when(icebergTable.schema()).thenReturn(schema); + Mockito.when(icebergTable.updateSchema()).thenReturn(updateSchema); + + try (MockedStatic mockedIcebergUtils = + Mockito.mockStatic(IcebergUtils.class, Mockito.CALLS_REAL_METHODS)) { + mockedIcebergUtils.when(() -> IcebergUtils.getIcebergTable(dorisTable)).thenReturn(icebergTable); + + ops.modifyColumn(dorisTable, ColumnPath.fromDotName("arr.element"), + new Column("element", Type.BIGINT, true), null, 1L); + ops.modifyColumn(dorisTable, ColumnPath.fromDotName("m.value"), + new Column("value", Type.BIGINT, true), null, 1L); + } + + Mockito.verify(updateSchema).updateColumn("arr.element", Types.LongType.get(), null); + Mockito.verify(updateSchema).updateColumn("m.value", Types.LongType.get(), null); + Mockito.verify(updateSchema, Mockito.never()).makeColumnOptional(Mockito.anyString()); + Mockito.verify(updateSchema, Mockito.times(2)).commit(); + } + + @Test + public void testModifyColumnRejectsPositionForDirectArrayElementAndMapValue() { + Schema schema = primitiveContainerSchema(); + ExternalTable dorisTable = Mockito.mock(ExternalTable.class); + Table icebergTable = Mockito.mock(Table.class); + Mockito.when(icebergTable.schema()).thenReturn(schema); + + try (MockedStatic mockedIcebergUtils = + Mockito.mockStatic(IcebergUtils.class, Mockito.CALLS_REAL_METHODS)) { + mockedIcebergUtils.when(() -> IcebergUtils.getIcebergTable(dorisTable)).thenReturn(icebergTable); + + assertUserException(() -> ops.modifyColumn(dorisTable, ColumnPath.fromDotName("arr.element"), + new Column("element", Type.BIGINT, true), ColumnPosition.FIRST, 1L), + "Cannot apply column position to 'arr.element': parent column path 'arr' is not a struct"); + assertUserException(() -> ops.modifyColumn(dorisTable, ColumnPath.fromDotName("arr.element"), + new Column("element", Type.BIGINT, true), new ColumnPosition("element"), 1L), + "Cannot apply column position to 'arr.element': parent column path 'arr' is not a struct"); + assertUserException(() -> ops.modifyColumn(dorisTable, ColumnPath.fromDotName("m.value"), + new Column("value", Type.BIGINT, true), ColumnPosition.FIRST, 1L), + "Cannot apply column position to 'm.value': parent column path 'm' is not a struct"); + assertUserException(() -> ops.modifyColumn(dorisTable, ColumnPath.fromDotName("m.value"), + new Column("value", Type.BIGINT, true), new ColumnPosition("value"), 1L), + "Cannot apply column position to 'm.value': parent column path 'm' is not a struct"); + } + + Mockito.verify(icebergTable, Mockito.never()).updateSchema(); + } + + @Test + public void testModifyColumnCommentUsesCanonicalNestedPaths() throws Throwable { + Schema schema = mixedCaseNestedSchema(); + ExternalTable dorisTable = Mockito.mock(ExternalTable.class); + Table icebergTable = Mockito.mock(Table.class); + UpdateSchema updateSchema = Mockito.mock(UpdateSchema.class); + Mockito.when(dorisTable.getRemoteDbName()).thenReturn("db"); + Mockito.when(icebergTable.schema()).thenReturn(schema); + Mockito.when(icebergTable.updateSchema()).thenReturn(updateSchema); + + try (MockedStatic mockedIcebergUtils = + Mockito.mockStatic(IcebergUtils.class, Mockito.CALLS_REAL_METHODS)) { + mockedIcebergUtils.when(() -> IcebergUtils.getIcebergTable(dorisTable)).thenReturn(icebergTable); + + ops.modifyColumnComment(dorisTable, ColumnPath.fromDotName("info.metric"), + "struct comment", 1L); + ops.modifyColumnComment(dorisTable, ColumnPath.fromDotName("events.element.score"), + "array element comment", 1L); + ops.modifyColumnComment(dorisTable, ColumnPath.fromDotName("attrs.value.code"), + "map value comment", 1L); + } + + Mockito.verify(updateSchema).updateColumnDoc("Info.Metric", "struct comment"); + Mockito.verify(updateSchema).updateColumnDoc("Events.element.Score", "array element comment"); + Mockito.verify(updateSchema).updateColumnDoc("Attrs.value.Code", "map value comment"); + Mockito.verify(updateSchema, Mockito.times(3)).commit(); + } + + @Test + public void testRejectsCommentsOnDirectArrayElementAndMapValue() { + Schema schema = primitiveContainerSchema(); + ExternalTable dorisTable = Mockito.mock(ExternalTable.class); + Table icebergTable = Mockito.mock(Table.class); + Mockito.when(dorisTable.getRemoteDbName()).thenReturn("db"); + Mockito.when(icebergTable.schema()).thenReturn(schema); + + try (MockedStatic mockedIcebergUtils = + Mockito.mockStatic(IcebergUtils.class, Mockito.CALLS_REAL_METHODS)) { + mockedIcebergUtils.when(() -> IcebergUtils.getIcebergTable(dorisTable)).thenReturn(icebergTable); + + assertUserException(() -> ops.modifyColumnComment( + dorisTable, ColumnPath.fromDotName("arr.element"), "array element comment", 1L), + "Iceberg does not support comments on collection element or value fields: arr.element"); + assertUserException(() -> ops.modifyColumnComment( + dorisTable, ColumnPath.fromDotName("m.value"), "map value comment", 1L), + "Iceberg does not support comments on collection element or value fields: m.value"); + assertUserException(() -> ops.modifyColumn(dorisTable, ColumnPath.fromDotName("arr.element"), + new Column("element", Type.BIGINT, true, "array element comment"), null, 1L), + "Iceberg does not support comments on collection element or value fields: arr.element"); + assertUserException(() -> ops.modifyColumn(dorisTable, ColumnPath.fromDotName("m.value"), + new Column("value", Type.BIGINT, true, "map value comment"), null, 1L), + "Iceberg does not support comments on collection element or value fields: m.value"); + Column arrayElementWithEmptyComment = new Column("element", Type.BIGINT, true, ""); + arrayElementWithEmptyComment.setCommentSpecified(true); + assertUserException(() -> ops.modifyColumn(dorisTable, ColumnPath.fromDotName("arr.element"), + arrayElementWithEmptyComment, null, 1L), + "Iceberg does not support comments on collection element or value fields: arr.element"); + Column mapValueWithEmptyComment = new Column("value", Type.BIGINT, true, ""); + mapValueWithEmptyComment.setCommentSpecified(true); + assertUserException(() -> ops.modifyColumn(dorisTable, ColumnPath.fromDotName("m.value"), + mapValueWithEmptyComment, null, 1L), + "Iceberg does not support comments on collection element or value fields: m.value"); + assertUserException(() -> ops.modifyColumnComment( + dorisTable, ColumnPath.fromDotName("m.key"), "map key comment", 1L), + "Cannot modify comment MAP key nested column"); + } + + Mockito.verify(icebergTable, Mockito.never()).updateSchema(); + } + + @Test + public void testRejectsTopLevelRowLineageMutationsForV3Tables() { + ExternalTable dorisTable = Mockito.mock(ExternalTable.class); + Table icebergTable = Mockito.mock(Table.class); + Mockito.when(icebergTable.properties()).thenReturn(Collections.singletonMap("format-version", "3")); + + try (MockedStatic mockedIcebergUtils = + Mockito.mockStatic(IcebergUtils.class, Mockito.CALLS_REAL_METHODS)) { + mockedIcebergUtils.when(() -> IcebergUtils.getIcebergTable(dorisTable)).thenReturn(icebergTable); + + assertUserException(() -> ops.addColumn(dorisTable, + new Column("_row_id", Type.BIGINT, true), null, 1L), + "Cannot add Iceberg v3 reserved row lineage column: _row_id"); + assertUserException(() -> ops.addColumns(dorisTable, Collections.singletonList( + new Column("_last_updated_sequence_number", Type.BIGINT, true)), 1L), + "Cannot add Iceberg v3 reserved row lineage column: _last_updated_sequence_number"); + assertUserException(() -> ops.dropColumn(dorisTable, "_ROW_ID", 1L), + "Cannot drop Iceberg v3 reserved row lineage column: _ROW_ID"); + assertUserException(() -> ops.renameColumn(dorisTable, "_row_id", "renamed", 1L), + "Cannot rename Iceberg v3 reserved row lineage column: _row_id"); + assertUserException(() -> ops.renameColumn( + dorisTable, "id", "_last_updated_sequence_number", 1L), + "Cannot rename to Iceberg v3 reserved row lineage column: _last_updated_sequence_number"); + assertUserException(() -> ops.modifyColumn(dorisTable, ColumnPath.of("_row_id"), + new Column("_row_id", Type.BIGINT, true), null, 1L), + "Cannot modify Iceberg v3 reserved row lineage column: _row_id"); + assertUserException(() -> ops.modifyColumnComment(dorisTable, + ColumnPath.of("_last_updated_sequence_number"), "comment", 1L), + "Cannot modify comment for Iceberg v3 reserved row lineage column: " + + "_last_updated_sequence_number"); + assertUserException(() -> ops.reorderColumns(dorisTable, + Arrays.asList("_row_id", "id"), 1L), + "Cannot reorder Iceberg v3 reserved row lineage column: _row_id"); + } + + Mockito.verify(icebergTable, Mockito.never()).updateSchema(); + } + + @Test + public void testAllowsV3NestedAndV2TopLevelRowLineageNames() throws Throwable { + Schema v3Schema = new Schema( + Types.NestedField.optional(1, "id", Types.IntegerType.get()), + Types.NestedField.optional(2, "s", Types.StructType.of( + Types.NestedField.optional(3, "source", Types.LongType.get()), + Types.NestedField.optional(4, "_row_id", Types.LongType.get())))); + ExternalTable v3DorisTable = Mockito.mock(ExternalTable.class); + Table v3IcebergTable = Mockito.mock(Table.class); + UpdateSchema v3UpdateSchema = Mockito.mock(UpdateSchema.class); + Mockito.when(v3DorisTable.getRemoteDbName()).thenReturn("db"); + Mockito.when(v3IcebergTable.properties()).thenReturn(Collections.singletonMap("format-version", "3")); + Mockito.when(v3IcebergTable.schema()).thenReturn(v3Schema); + Mockito.when(v3IcebergTable.updateSchema()).thenReturn(v3UpdateSchema); + + Schema v2Schema = new Schema(Types.NestedField.optional(1, "id", Types.IntegerType.get())); + ExternalTable v2DorisTable = Mockito.mock(ExternalTable.class); + Table v2IcebergTable = Mockito.mock(Table.class); + UpdateSchema v2UpdateSchema = Mockito.mock(UpdateSchema.class); + Mockito.when(v2DorisTable.getRemoteDbName()).thenReturn("db"); + Mockito.when(v2IcebergTable.properties()).thenReturn(Collections.singletonMap("format-version", "2")); + Mockito.when(v2IcebergTable.schema()).thenReturn(v2Schema); + Mockito.when(v2IcebergTable.updateSchema()).thenReturn(v2UpdateSchema); + + try (MockedStatic mockedIcebergUtils = + Mockito.mockStatic(IcebergUtils.class, Mockito.CALLS_REAL_METHODS)) { + mockedIcebergUtils.when(() -> IcebergUtils.getIcebergTable(v3DorisTable)).thenReturn(v3IcebergTable); + mockedIcebergUtils.when(() -> IcebergUtils.getIcebergTable(v2DorisTable)).thenReturn(v2IcebergTable); + + ops.addColumn(v3DorisTable, ColumnPath.fromDotName("s._last_updated_sequence_number"), + new Column("_last_updated_sequence_number", Type.BIGINT, true), null, 1L); + ops.renameColumn(v3DorisTable, ColumnPath.fromDotName("s.source"), "_last_updated_sequence_number", 1L); + ops.modifyColumn(v3DorisTable, ColumnPath.fromDotName("s._row_id"), + new Column("_row_id", Type.BIGINT, true), null, 1L); + ops.modifyColumnComment(v3DorisTable, ColumnPath.fromDotName("s._row_id"), "comment", 1L); + ops.dropColumn(v3DorisTable, ColumnPath.fromDotName("s._row_id"), 1L); + + ops.addColumn(v2DorisTable, new Column("_row_id", Type.BIGINT, true), null, 1L); + ops.renameColumn(v2DorisTable, "id", "_last_updated_sequence_number", 1L); + } + + Mockito.verify(v3UpdateSchema, Mockito.times(5)).commit(); + Mockito.verify(v2UpdateSchema, Mockito.times(2)).commit(); + } + + @Test + public void testResolveNestedColumnPathRejectsMapKey() { + assertUserException(() -> ops.resolveNestedColumnPath(nestedSchema(), ColumnPath.fromDotName("m.key.x"), + "modify"), + "Cannot modify MAP key nested column"); + } + + @Test + public void testResolveNestedColumnPathRejectsPrimitiveParent() { + assertUserException(() -> ops.resolveNestedColumnPath(nestedSchema(), ColumnPath.fromDotName("id.x"), + "modify"), + "Cannot resolve nested field under primitive column path"); + } + + @Test + public void testGetPositionReferencePathForNestedColumn() { + Assert.assertEquals("s.a", ops.getPositionReferencePath(ColumnPath.fromDotName("s.new_col"), + new ColumnPosition("a"))); + Assert.assertEquals("arr.element.x", ops.getPositionReferencePath( + ColumnPath.fromDotName("arr.element.new_col"), new ColumnPosition("x"))); + Assert.assertEquals("m.value.v", ops.getPositionReferencePath( + ColumnPath.fromDotName("m.value.new_col"), new ColumnPosition("v"))); + Assert.assertEquals("id", ops.getPositionReferencePath(ColumnPath.fromDotName("new_col"), + new ColumnPosition("id"))); + } + + @Test + public void testValidateNestedStructFieldSupportsStructArrayElementAndMapValueFields() throws Throwable { + Schema schema = nestedSchema(); + Assert.assertTrue(ops.validateNestedStructField(schema, ColumnPath.fromDotName("s.a"), "drop") + .isPrimitiveType()); + Assert.assertTrue(ops.validateNestedStructField(schema, ColumnPath.fromDotName("arr.element.x"), "drop") + .isPrimitiveType()); + Assert.assertTrue(ops.validateNestedStructField(schema, ColumnPath.fromDotName("m.value.v"), "rename") + .isPrimitiveType()); + } + + @Test + public void testValidateNestedStructFieldRejectsArrayElementAndMapValuePseudoFields() { + assertUserException(() -> ops.validateNestedStructField(nestedSchema(), ColumnPath.fromDotName("arr.element"), + "drop"), + "Parent column path 'arr' is not a struct"); + assertUserException(() -> ops.validateNestedStructField(nestedSchema(), ColumnPath.fromDotName("m.value"), + "rename"), + "Parent column path 'm' is not a struct"); + } + + @Test + public void testValidateNestedStructFieldRejectsMapKeyAndMissingField() { + assertUserException(() -> ops.validateNestedStructField(nestedSchema(), ColumnPath.fromDotName("m.key.k"), + "drop"), + "Cannot drop MAP key nested column"); + assertUserException(() -> ops.validateNestedStructField(nestedSchema(), ColumnPath.fromDotName("s.missing"), + "rename"), + "Column path does not exist in Iceberg schema"); + } + private void invokeValidateForModifyColumn(Column column, NestedField currentCol) throws Throwable { invokeValidationMethod(validateForModifyColumnMethod, column, currentCol); } @@ -203,8 +1424,9 @@ private void assertUserException(ThrowingRunnable runnable, String expectedMessa runnable.run(); Assert.fail("expected UserException"); } catch (Throwable t) { - Assert.assertTrue(t instanceof UserException); - Assert.assertTrue(t.getMessage().contains(expectedMessage)); + Assert.assertTrue("expected UserException but got " + t, t instanceof UserException); + Assert.assertTrue("expected message containing '" + expectedMessage + "' but was '" + + t.getMessage() + "'", t.getMessage().contains(expectedMessage)); } } @@ -212,4 +1434,86 @@ private void assertUserException(ThrowingRunnable runnable, String expectedMessa private interface ThrowingRunnable { void run() throws Throwable; } + + private Schema nestedSchema() { + return new Schema( + Types.NestedField.optional(1, "id", Types.IntegerType.get()), + Types.NestedField.optional(2, "s", Types.StructType.of( + Types.NestedField.optional(3, "a", Types.IntegerType.get()))), + Types.NestedField.optional(4, "arr", Types.ListType.ofOptional(5, + Types.StructType.of(Types.NestedField.optional(6, "x", Types.IntegerType.get())))), + Types.NestedField.optional(7, "m", Types.MapType.ofOptional(8, 9, + Types.StringType.get(), + Types.StructType.of(Types.NestedField.optional(10, "v", Types.IntegerType.get()))))); + } + + private Schema mixedCaseNestedSchema() { + return new Schema( + Types.NestedField.optional(1, "Id", Types.IntegerType.get()), + Types.NestedField.optional(2, "Info", Types.StructType.of( + Types.NestedField.optional(3, "Metric", Types.IntegerType.get()), + Types.NestedField.optional(4, "Label", Types.StringType.get()))), + Types.NestedField.optional(5, "Events", Types.ListType.ofOptional(6, + Types.StructType.of(Types.NestedField.optional(7, "Score", Types.IntegerType.get())))), + Types.NestedField.optional(8, "Attrs", Types.MapType.ofOptional(9, 10, + Types.StringType.get(), + Types.StructType.of(Types.NestedField.optional(11, "Code", Types.IntegerType.get()))))); + } + + private Schema primitiveContainerSchema() { + return new Schema( + Types.NestedField.optional(1, "arr", + Types.ListType.ofOptional(2, Types.IntegerType.get())), + Types.NestedField.optional(3, "m", Types.MapType.ofOptional( + 4, 5, Types.StringType.get(), Types.IntegerType.get()))); + } + + private Schema mappedPrimitiveSchema() { + return new Schema( + Types.NestedField.required(1, "top_uuid", Types.UUIDType.get()), + Types.NestedField.required(2, "top_other", Types.IntegerType.get()), + Types.NestedField.optional(3, "info", Types.StructType.of( + Types.NestedField.required(4, "uuid_value", Types.UUIDType.get()), + Types.NestedField.required(5, "tz_value", Types.TimestampType.withZone()), + Types.NestedField.required(6, "other", Types.IntegerType.get())))); + } + + private Schema mappedComplexSchema() { + return new Schema(Types.NestedField.optional(1, "outer", Types.StructType.of( + Types.NestedField.optional(2, "payload", Types.StructType.of( + Types.NestedField.optional(3, "uuid_value", Types.UUIDType.get()), + Types.NestedField.optional(4, "binary_value", Types.BinaryType.get()), + Types.NestedField.optional(5, "fixed_value", Types.FixedType.ofLength(8)), + Types.NestedField.optional(6, "tz_value", Types.TimestampType.withZone()), + Types.NestedField.optional(7, "metric", Types.IntegerType.get()), + Types.NestedField.optional(8, "events", + Types.ListType.ofOptional(9, Types.UUIDType.get())), + Types.NestedField.optional(10, "attrs", Types.MapType.ofOptional( + 11, 12, Types.FixedType.ofLength(4), Types.TimestampType.withZone()))))))); + } + + private StructType mappedPayloadDorisType(Type metricType, int fixedLength, Type mapKeyType) { + return new StructType( + new StructField("uuid_value", ScalarType.createVarbinaryType(16)), + new StructField("binary_value", + ScalarType.createVarbinaryType(ScalarType.MAX_VARBINARY_LENGTH)), + new StructField("fixed_value", ScalarType.createVarbinaryType(fixedLength)), + new StructField("tz_value", ScalarType.createTimeStampTzType(6)), + new StructField("metric", metricType), + new StructField("events", ArrayType.create( + ScalarType.createVarbinaryType(16), true)), + new StructField("attrs", new MapType( + mapKeyType, ScalarType.createTimeStampTzType(6)))); + } + + private Schema requiredNestedSchema() { + return new Schema(Types.NestedField.required(1, "info", Types.StructType.of( + Types.NestedField.required(2, "metric", Types.IntegerType.get()), + Types.NestedField.required(3, "child", Types.StructType.of( + Types.NestedField.required(4, "value", Types.IntegerType.get()))), + Types.NestedField.required(5, "events", Types.ListType.ofRequired( + 6, Types.IntegerType.get())), + Types.NestedField.required(7, "attrs", Types.MapType.ofRequired( + 8, 9, Types.StringType.get(), Types.IntegerType.get()))))); + } } diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/paimon/source/PaimonScanNodeTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/paimon/source/PaimonScanNodeTest.java index 9dcf91731bbee2..595f64758dfe97 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/datasource/paimon/source/PaimonScanNodeTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/paimon/source/PaimonScanNodeTest.java @@ -35,6 +35,7 @@ import org.apache.doris.qe.SessionVariable; import org.apache.doris.thrift.TFileRangeDesc; import org.apache.doris.thrift.TFileScanRangeParams; +import org.apache.doris.thrift.TPaimonReaderType; import org.apache.doris.thrift.TPushAggOp; import org.apache.paimon.data.BinaryRow; @@ -736,6 +737,26 @@ public void testNativeSplitCarriesPartitionMetadataWithoutRuntimeFilterPruning() Assert.assertEquals(Collections.emptyList(), split.getPartitionValues()); } + @Test + public void testSetPaimonParamsUsesJniWhenCppOptionEnabled() throws Exception { + Mockito.when(sv.isEnablePaimonCppReader()).thenReturn(true); + PaimonScanNode node = newTestNode(new PlanNodeId(0), new TupleId(0), sv); + PaimonSource source = Mockito.mock(PaimonSource.class); + Mockito.when(source.getTableLocation()).thenReturn("file:///warehouse"); + Table paimonTable = mockPaimonTableWithPartitionKeys(Collections.emptyList()); + Mockito.when(source.getPaimonTable()).thenReturn(paimonTable); + node.setSource(source); + + TFileRangeDesc rangeDesc = new TFileRangeDesc(); + invokePrivateMethod(node, "setPaimonParams", + new Class[] {TFileRangeDesc.class, PaimonSplit.class}, + rangeDesc, new PaimonSplit(createDataSplit("jni-only.parquet"))); + + Assert.assertEquals(TPaimonReaderType.PAIMON_JNI, + rangeDesc.getTableFormatParams().getPaimonParams().getReaderType()); + Assert.assertTrue(rangeDesc.getTableFormatParams().getPaimonParams().isSetPaimonSplit()); + } + @Test public void testGetFieldIndexMatchesMixedCaseColumns() { List fieldNames = Arrays.asList("data", "mIxEd_COL", "PART"); @@ -753,6 +774,12 @@ private PaimonScanNode newTestNode(PlanNodeId id, TupleId tupleId, SessionVariab return new PaimonScanNode(id, new TupleDescriptor(tupleId), false, sessionVariable, ScanContext.EMPTY); } + private Table mockPaimonTableWithPartitionKeys(List partitionKeys) { + Table paimonTable = Mockito.mock(Table.class); + Mockito.when(paimonTable.partitionKeys()).thenReturn(partitionKeys); + return paimonTable; + } + private void mockNativeReader(PaimonScanNode spyNode) { Mockito.doReturn(true).when(spyNode).supportNativeReader(ArgumentMatchers.any(Optional.class)); } diff --git a/fe/fe-core/src/test/java/org/apache/doris/nereids/parser/IcebergNestedSchemaEvolutionParserTest.java b/fe/fe-core/src/test/java/org/apache/doris/nereids/parser/IcebergNestedSchemaEvolutionParserTest.java new file mode 100644 index 00000000000000..9ff08e07a0455f --- /dev/null +++ b/fe/fe-core/src/test/java/org/apache/doris/nereids/parser/IcebergNestedSchemaEvolutionParserTest.java @@ -0,0 +1,463 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.nereids.parser; + +import org.apache.doris.analysis.ColumnPath; +import org.apache.doris.nereids.exceptions.ParseException; +import org.apache.doris.nereids.trees.plans.Plan; +import org.apache.doris.nereids.trees.plans.commands.AlterTableCommand; +import org.apache.doris.nereids.trees.plans.commands.info.AddColumnOp; +import org.apache.doris.nereids.trees.plans.commands.info.AddColumnsOp; +import org.apache.doris.nereids.trees.plans.commands.info.AlterTableOp; +import org.apache.doris.nereids.trees.plans.commands.info.DropColumnOp; +import org.apache.doris.nereids.trees.plans.commands.info.ModifyColumnCommentOp; +import org.apache.doris.nereids.trees.plans.commands.info.ModifyColumnOp; +import org.apache.doris.nereids.trees.plans.commands.info.RenameColumnOp; +import org.apache.doris.nereids.types.StructType; +import org.apache.doris.qe.SqlModeHelper; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; +import org.mockito.MockedStatic; +import org.mockito.Mockito; + +import java.util.Arrays; +import java.util.Collections; +import java.util.List; +import java.util.Locale; + +public class IcebergNestedSchemaEvolutionParserTest { + + private final NereidsParser parser = new NereidsParser(); + + @Test + public void testParseNestedAddColumnPaths() { + assertSingleClausePath("ALTER TABLE t ADD COLUMN s.c STRING NULL", + AddColumnOp.class, "s.c"); + assertSingleClausePath("ALTER TABLE t ADD COLUMN s.first_col INT NULL FIRST", + AddColumnOp.class, "s.first_col"); + assertSingleClausePath("ALTER TABLE t ADD COLUMN s.after_col INT NULL AFTER a", + AddColumnOp.class, "s.after_col"); + assertSingleClausePath("ALTER TABLE t ADD COLUMN arr.element.y INT NULL", + AddColumnOp.class, "arr.element.y"); + assertSingleClausePath("ALTER TABLE t ADD COLUMN m.value.y INT NULL", + AddColumnOp.class, "m.value.y"); + } + + @Test + public void testParseNestedModifyDropRenamePaths() { + assertSingleClausePath("ALTER TABLE t MODIFY COLUMN s.a BIGINT", + ModifyColumnOp.class, "s.a"); + assertSingleClausePath("ALTER TABLE t MODIFY COLUMN s.a BIGINT AFTER b", + ModifyColumnOp.class, "s.a"); + assertSingleClausePath("ALTER TABLE t MODIFY COLUMN arr.element BIGINT", + ModifyColumnOp.class, "arr.element"); + assertSingleClausePath("ALTER TABLE t MODIFY COLUMN m.value BIGINT", + ModifyColumnOp.class, "m.value"); + assertSingleClausePath("ALTER TABLE t MODIFY COLUMN s.a COMMENT 'nested comment'", + ModifyColumnCommentOp.class, "s.a"); + assertSingleClausePath("ALTER TABLE t DROP COLUMN s.c", + DropColumnOp.class, "s.c"); + assertSingleClausePath("ALTER TABLE t DROP COLUMN arr.element.y", + DropColumnOp.class, "arr.element.y"); + assertSingleClausePath("ALTER TABLE t DROP COLUMN m.value.y", + DropColumnOp.class, "m.value.y"); + assertSingleClausePath("ALTER TABLE t RENAME COLUMN s.c TO c2", + RenameColumnOp.class, "s.c"); + assertSingleClausePath("ALTER TABLE t RENAME COLUMN arr.element.y TO y2", + RenameColumnOp.class, "arr.element.y"); + assertSingleClausePath("ALTER TABLE t RENAME COLUMN m.value.y TO y2", + RenameColumnOp.class, "m.value.y"); + } + + @Test + public void testNestedColumnDefaultClausesAreNotInGrammar() { + for (String sql : Arrays.asList( + "ALTER TABLE t ADD COLUMN s.b BIGINT NULL DEFAULT 7", + "ALTER TABLE t ADD COLUMN s.c BIGINT NULL DEFAULT NULL", + "ALTER TABLE t ADD COLUMN s.ts DATETIME NULL DEFAULT CURRENT_TIMESTAMP " + + "ON UPDATE CURRENT_TIMESTAMP", + "ALTER TABLE t MODIFY COLUMN s.a BIGINT DEFAULT 7", + "ALTER TABLE t MODIFY COLUMN s.a BIGINT DEFAULT NULL", + "ALTER TABLE t MODIFY COLUMN s.ts DATETIME DEFAULT CURRENT_TIMESTAMP " + + "ON UPDATE CURRENT_TIMESTAMP")) { + Assertions.assertThrows(ParseException.class, () -> parser.parseSingle(sql), sql); + } + } + + @Test + public void testTopLevelColumnKeepsExistingDefaultGrammar() { + AddColumnOp add = assertSingleClausePath( + "ALTER TABLE t ADD COLUMN b BIGINT NULL DEFAULT 7", AddColumnOp.class, "b"); + Assertions.assertTrue(add.getColumnDef().hasDefaultValue()); + } + + @Test + public void testTopLevelColumnRoundTripPreservesOmittedIntent() { + AddColumnOp add = assertSingleClausePath( + "ALTER TABLE t ADD COLUMN added INT", AddColumnOp.class, "added"); + String renderedAdd = add.toSql(); + Assertions.assertFalse(renderedAdd.contains(" NULL")); + Assertions.assertFalse(renderedAdd.contains(" COMMENT ")); + AddColumnOp reparsedAdd = assertSingleClausePath( + "ALTER TABLE t " + renderedAdd, AddColumnOp.class, "added"); + Assertions.assertFalse(reparsedAdd.getColumnDef() + .translateToCatalogStyleForSchemaChange().isNullableSpecified()); + Assertions.assertFalse(reparsedAdd.getColumnDef() + .translateToCatalogStyleForSchemaChange().isCommentSpecified()); + + ModifyColumnOp modify = assertSingleClausePath( + "ALTER TABLE t MODIFY COLUMN existing BIGINT", ModifyColumnOp.class, "existing"); + String renderedModify = modify.toSql(); + Assertions.assertFalse(renderedModify.contains(" NULL")); + Assertions.assertFalse(renderedModify.contains(" COMMENT ")); + ModifyColumnOp reparsedModify = assertSingleClausePath( + "ALTER TABLE t " + renderedModify, ModifyColumnOp.class, "existing"); + Assertions.assertFalse(reparsedModify.getColumnDef() + .translateToCatalogStyleForSchemaChange().isNullableSpecified()); + Assertions.assertFalse(reparsedModify.getColumnDef() + .translateToCatalogStyleForSchemaChange().isCommentSpecified()); + } + + @Test + public void testLegacyStringConstructorsKeepDottedTopLevelNames() { + DropColumnOp drop = new DropColumnOp("top.level", null, Collections.emptyMap()); + RenameColumnOp rename = new RenameColumnOp("top.level", "renamed"); + ModifyColumnCommentOp comment = new ModifyColumnCommentOp("top.level", "comment"); + + Assertions.assertFalse(drop.getColumnPath().isNested()); + Assertions.assertFalse(rename.getColumnPath().isNested()); + Assertions.assertFalse(comment.getColumnPath().isNested()); + Assertions.assertEquals("top.level", drop.getColumnPath().getFullPath()); + Assertions.assertEquals("top.level", rename.getColumnPath().getFullPath()); + Assertions.assertEquals("top.level", comment.getColumnPath().getFullPath()); + } + + @Test + public void testQuotedNestedIdentifiersAreNormalized() { + ModifyColumnOp dottedTopLevel = assertSingleClausePath( + "ALTER TABLE t MODIFY COLUMN `a.b` BIGINT", + ModifyColumnOp.class, "a.b"); + Assertions.assertFalse(dottedTopLevel.getColumnPath().isNested()); + + assertSingleClausePath("ALTER TABLE t MODIFY COLUMN info.`Metric` BIGINT", + ModifyColumnOp.class, "info.Metric"); + assertSingleClausePath("ALTER TABLE t MODIFY COLUMN m_scalar.`key` BIGINT", + ModifyColumnOp.class, "m_scalar.key"); + assertSingleClausePath("ALTER TABLE t MODIFY COLUMN info.`Metric``Name` COMMENT 'quoted'", + ModifyColumnCommentOp.class, "info.Metric`Name"); + + AddColumnOp add = assertSingleClausePath( + "ALTER TABLE t ADD COLUMN info.`New``Field` INT NULL AFTER `Old``Field`", + AddColumnOp.class, "info.New`Field"); + Assertions.assertEquals("Old`Field", add.getColPos().getLastCol()); + AddColumnOp reparsedAdd = assertSingleClausePath( + "ALTER TABLE t " + add.toSql(), AddColumnOp.class, "info.New`Field"); + Assertions.assertEquals("Old`Field", reparsedAdd.getColPos().getLastCol()); + + RenameColumnOp rename = assertSingleClausePath( + "ALTER TABLE t RENAME COLUMN info.`Metric``Name` TO `New``Metric`", + RenameColumnOp.class, "info.Metric`Name"); + Assertions.assertEquals("New`Metric", rename.getNewColName()); + RenameColumnOp reparsedRename = assertSingleClausePath( + "ALTER TABLE t " + rename.toSql(), RenameColumnOp.class, "info.Metric`Name"); + Assertions.assertEquals("New`Metric", reparsedRename.getNewColName()); + } + + @Test + public void testStructMemberIdentifiersRoundTrip() { + AddColumnOp add = assertSingleClausePath( + "ALTER TABLE t ADD COLUMN info.payload " + + "STRUCT<`key`:INT,`Metric``Name`:STRING> NULL", + AddColumnOp.class, "info.payload"); + assertStructMemberNames((StructType) add.getColumnDef().getType()); + Assertions.assertTrue(add.toSql().contains("STRUCT<`key`:INT,`metric``name`:TEXT>")); + AddColumnOp reparsedAdd = assertSingleClausePath( + "ALTER TABLE t " + add.toSql(), AddColumnOp.class, "info.payload"); + assertStructMemberNames((StructType) reparsedAdd.getColumnDef().getType()); + + ModifyColumnOp modify = assertSingleClausePath( + "ALTER TABLE t MODIFY COLUMN info.payload " + + "STRUCT<`key`:INT,`Metric``Name`:STRING>", + ModifyColumnOp.class, "info.payload"); + assertStructMemberNames((StructType) modify.getColumnDef().getType()); + ModifyColumnOp reparsedModify = assertSingleClausePath( + "ALTER TABLE t " + modify.toSql(), ModifyColumnOp.class, "info.payload"); + assertStructMemberNames((StructType) reparsedModify.getColumnDef().getType()); + + ModifyColumnOp ordinary = assertSingleClausePath( + "ALTER TABLE t MODIFY COLUMN info.payload STRUCT", + ModifyColumnOp.class, "info.payload"); + Assertions.assertTrue(ordinary.toSql().contains("STRUCT")); + } + + @Test + public void testStructFieldSqlRoundTripIsLocaleIndependent() { + Locale originalLocale = Locale.getDefault(); + try { + Locale.setDefault(Locale.forLanguageTag("tr-TR")); + StructType structType = (StructType) parser.parseDataType("STRUCT<`in`:INT>"); + String renderedType = structType.toSql(); + + Assertions.assertTrue(renderedType.contains("`in`:INT")); + Assertions.assertDoesNotThrow(() -> parser.parseDataType(renderedType)); + } finally { + Locale.setDefault(originalLocale); + } + } + + @Test + public void testEmptyQuotedIdentifiersAreRejectedAsParseErrors() { + for (String sql : Arrays.asList( + "ALTER TABLE t ADD COLUMN `` INT NULL", + "ALTER TABLE t ADD COLUMN info.`` INT NULL", + "ALTER TABLE t MODIFY COLUMN info.`` BIGINT", + "ALTER TABLE t MODIFY COLUMN info.`` COMMENT 'comment'", + "ALTER TABLE t DROP COLUMN info.``", + "ALTER TABLE t RENAME COLUMN info.`` TO renamed", + "ALTER TABLE t ORDER BY (id, ``)")) { + ParseException exception = Assertions.assertThrows(ParseException.class, + () -> parser.parseSingle(sql), sql); + Assertions.assertTrue(exception.getMessage().contains("Quoted identifier cannot be empty"), sql); + } + } + + @Test + public void testEmptyQuotedIdentifiersOutsideColumnPathsKeepExistingParserSemantics() { + Assertions.assertDoesNotThrow(() -> parser.parseSingle("ALTER TABLE t CREATE BRANCH ``")); + Assertions.assertDoesNotThrow( + () -> parser.parseSingle("GRANT SELECT_PRIV ON `internal`.``.`` TO 'user1'")); + } + + @Test + public void testModifyColumnRoundTripPreservesNullabilityIntent() { + ModifyColumnOp omitted = assertSingleClausePath( + "ALTER TABLE t MODIFY COLUMN info.metric BIGINT", + ModifyColumnOp.class, "info.metric"); + Assertions.assertFalse(omitted.getColumnDef() + .translateToCatalogStyleForSchemaChange().isNullableSpecified()); + Assertions.assertFalse(omitted.toSql().contains(" BIGINT NULL ")); + + ModifyColumnOp reparsedOmitted = assertSingleClausePath( + "ALTER TABLE t " + omitted.toSql(), ModifyColumnOp.class, "info.metric"); + Assertions.assertFalse(reparsedOmitted.getColumnDef() + .translateToCatalogStyleForSchemaChange().isNullableSpecified()); + + ModifyColumnOp nullable = assertSingleClausePath( + "ALTER TABLE t MODIFY COLUMN info.metric BIGINT NULL", + ModifyColumnOp.class, "info.metric"); + ModifyColumnOp reparsedNullable = assertSingleClausePath( + "ALTER TABLE t " + nullable.toSql(), ModifyColumnOp.class, "info.metric"); + Assertions.assertTrue(reparsedNullable.getColumnDef() + .translateToCatalogStyleForSchemaChange().isNullableSpecified()); + } + + @Test + public void testModifyColumnRoundTripPreservesCommentIntent() { + ModifyColumnOp omitted = assertSingleClausePath( + "ALTER TABLE t MODIFY COLUMN arr.element BIGINT", + ModifyColumnOp.class, "arr.element"); + Assertions.assertFalse(omitted.getColumnDef() + .translateToCatalogStyleForSchemaChange().isCommentSpecified()); + Assertions.assertFalse(omitted.toSql().contains(" COMMENT ")); + + ModifyColumnOp reparsedOmitted = assertSingleClausePath( + "ALTER TABLE t " + omitted.toSql(), ModifyColumnOp.class, "arr.element"); + Assertions.assertFalse(reparsedOmitted.getColumnDef() + .translateToCatalogStyleForSchemaChange().isCommentSpecified()); + + ModifyColumnOp empty = assertSingleClausePath( + "ALTER TABLE t MODIFY COLUMN arr.element BIGINT COMMENT ''", + ModifyColumnOp.class, "arr.element"); + ModifyColumnOp reparsedEmpty = assertSingleClausePath( + "ALTER TABLE t " + empty.toSql(), ModifyColumnOp.class, "arr.element"); + Assertions.assertTrue(reparsedEmpty.getColumnDef() + .translateToCatalogStyleForSchemaChange().isCommentSpecified()); + Assertions.assertEquals("", reparsedEmpty.getColumnDef().getComment()); + } + + @Test + public void testStructMemberRoundTripPreservesCommentIntent() { + ModifyColumnOp modify = assertSingleClausePath( + "ALTER TABLE t MODIFY COLUMN info.payload " + + "STRUCT", + ModifyColumnOp.class, "info.payload"); + StructType structType = (StructType) modify.getColumnDef().getType(); + Assertions.assertFalse(structType.getFields().get(0).isCommentSpecified()); + Assertions.assertTrue(structType.getFields().get(1).isCommentSpecified()); + + ModifyColumnOp reparsed = assertSingleClausePath( + "ALTER TABLE t " + modify.toSql(), ModifyColumnOp.class, "info.payload"); + StructType reparsedType = (StructType) reparsed.getColumnDef().getType(); + Assertions.assertFalse(reparsedType.getFields().get(0).isCommentSpecified()); + Assertions.assertTrue(reparsedType.getFields().get(1).isCommentSpecified()); + Assertions.assertEquals("", reparsedType.getFields().get(1).getComment()); + org.apache.doris.catalog.StructType catalogType = (org.apache.doris.catalog.StructType) reparsed + .getColumnDef().translateToCatalogStyleForSchemaChange().getType(); + Assertions.assertFalse(catalogType.getFields().get(0).isCommentSpecified()); + Assertions.assertTrue(catalogType.getFields().get(1).isCommentSpecified()); + } + + @Test + public void testModifyColumnCommentRoundTripEscapesQuotesAndBackslashes() { + assertCommentRoundTrip(false); + assertCommentRoundTrip(true); + } + + @Test + public void testColumnDefinitionCommentRoundTripEscapesQuotesAndBackslashes() { + assertColumnDefinitionCommentRoundTrip(false); + assertColumnDefinitionCommentRoundTrip(true); + } + + @Test + public void testRegularColumnDefinitionCommentRoundTripEscapesQuotesAndBackslashes() { + assertRegularColumnDefinitionCommentRoundTrip(false); + assertRegularColumnDefinitionCommentRoundTrip(true); + } + + @Test + public void testStructMemberCommentRoundTripEscapesQuotesAndBackslashes() { + assertStructMemberCommentRoundTrip(false); + assertStructMemberCommentRoundTrip(true); + } + + private void assertStructMemberCommentRoundTrip(boolean noBackslashEscapes) { + try (MockedStatic mockedSqlMode = Mockito.mockStatic(SqlModeHelper.class)) { + mockedSqlMode.when(SqlModeHelper::hasNoBackSlashEscapes).thenReturn(noBackslashEscapes); + + String sqlPath = noBackslashEscapes ? "C:\\tmp\\" : "C:\\\\tmp\\\\"; + String expectedComment = "owner's \"field\" C:\\tmp\\"; + ModifyColumnOp modify = assertSingleClausePath( + "ALTER TABLE t MODIFY COLUMN info.payload " + + "STRUCT", + ModifyColumnOp.class, "info.payload"); + StructType structType = (StructType) modify.getColumnDef().getType(); + Assertions.assertEquals(expectedComment, structType.getFields().get(0).getComment()); + + ModifyColumnOp reparsedModify = assertSingleClausePath( + "ALTER TABLE t " + modify.toSql(), ModifyColumnOp.class, "info.payload"); + StructType reparsedType = (StructType) reparsedModify.getColumnDef().getType(); + Assertions.assertEquals(expectedComment, reparsedType.getFields().get(0).getComment()); + } + } + + private void assertStructMemberNames(StructType structType) { + Assertions.assertEquals("key", structType.getFields().get(0).getName()); + Assertions.assertEquals("metric`name", structType.getFields().get(1).getName()); + } + + private void assertRegularColumnDefinitionCommentRoundTrip(boolean noBackslashEscapes) { + try (MockedStatic mockedSqlMode = Mockito.mockStatic(SqlModeHelper.class)) { + mockedSqlMode.when(SqlModeHelper::hasNoBackSlashEscapes).thenReturn(noBackslashEscapes); + + String sqlPath = noBackslashEscapes ? "C:\\tmp\\" : "C:\\\\tmp\\\\"; + String expectedComment = "owner's \"field\" C:\\tmp\\"; + AddColumnsOp addColumns = assertSingleClause( + "ALTER TABLE t ADD COLUMN (owner STRING NULL COMMENT 'owner''s \"field\" " + + sqlPath + "', metric INT NULL)", AddColumnsOp.class); + Assertions.assertEquals(expectedComment, + addColumns.getColumnDefinitions().get(0).getComment()); + + AddColumnsOp reparsedAddColumns = assertSingleClause( + "ALTER TABLE t " + addColumns.toSql(), AddColumnsOp.class); + Assertions.assertEquals(expectedComment, + reparsedAddColumns.getColumnDefinitions().get(0).getComment()); + } + } + + private void assertColumnDefinitionCommentRoundTrip(boolean noBackslashEscapes) { + try (MockedStatic mockedSqlMode = Mockito.mockStatic(SqlModeHelper.class)) { + mockedSqlMode.when(SqlModeHelper::hasNoBackSlashEscapes).thenReturn(noBackslashEscapes); + + String sqlPath = noBackslashEscapes ? "C:\\tmp\\" : "C:\\\\tmp\\\\"; + String expectedComment = "owner's \"field\" C:\\tmp\\"; + AddColumnOp add = assertSingleClausePath( + "ALTER TABLE t ADD COLUMN info.owner STRING NULL COMMENT 'owner''s \"field\" " + + sqlPath + "'", + AddColumnOp.class, "info.owner"); + Assertions.assertEquals(expectedComment, add.getColumnDef().getComment()); + AddColumnOp reparsedAdd = assertSingleClausePath( + "ALTER TABLE t " + add.toSql(), AddColumnOp.class, "info.owner"); + Assertions.assertEquals(expectedComment, reparsedAdd.getColumnDef().getComment()); + + ModifyColumnOp modify = assertSingleClausePath( + "ALTER TABLE t MODIFY COLUMN info.owner STRING COMMENT 'owner''s \"field\" " + + sqlPath + "'", + ModifyColumnOp.class, "info.owner"); + Assertions.assertEquals(expectedComment, modify.getColumnDef().getComment()); + ModifyColumnOp reparsedModify = assertSingleClausePath( + "ALTER TABLE t " + modify.toSql(), ModifyColumnOp.class, "info.owner"); + Assertions.assertEquals(expectedComment, reparsedModify.getColumnDef().getComment()); + } + } + + private void assertCommentRoundTrip(boolean noBackslashEscapes) { + try (MockedStatic mockedSqlMode = Mockito.mockStatic(SqlModeHelper.class)) { + mockedSqlMode.when(SqlModeHelper::hasNoBackSlashEscapes).thenReturn(noBackslashEscapes); + + String expectedComment = "owner's \"field\" C:\\tmp\\"; + ModifyColumnCommentOp comment = new ModifyColumnCommentOp( + ColumnPath.fromDotName("info.metric"), expectedComment); + String renderedSql = comment.toSql(); + Assertions.assertTrue(renderedSql.contains("\"\"field\"\"")); + Assertions.assertTrue(renderedSql.contains(noBackslashEscapes + ? "C:\\tmp\\" : "C:\\\\tmp\\\\")); + + ModifyColumnCommentOp reparsed = assertSingleClausePath( + "ALTER TABLE t " + renderedSql, ModifyColumnCommentOp.class, "info.metric"); + Assertions.assertEquals(expectedComment, reparsed.getComment()); + + ModifyColumnCommentOp doubledSingleQuote = assertSingleClausePath( + "ALTER TABLE t MODIFY COLUMN info.metric COMMENT 'owner''s'", + ModifyColumnCommentOp.class, "info.metric"); + Assertions.assertEquals("owner's", doubledSingleQuote.getComment()); + ModifyColumnCommentOp doubledDoubleQuote = assertSingleClausePath( + "ALTER TABLE t MODIFY COLUMN info.metric COMMENT \"a\"\"b\"", + ModifyColumnCommentOp.class, "info.metric"); + Assertions.assertEquals("a\"b", doubledDoubleQuote.getComment()); + } + } + + private T assertSingleClausePath(String sql, Class clauseClass, + String expectedPath) { + T clause = assertSingleClause(sql, clauseClass); + if (clause instanceof AddColumnOp) { + Assertions.assertEquals(expectedPath, ((AddColumnOp) clause).getColumnPath().getFullPath()); + } else if (clause instanceof ModifyColumnOp) { + Assertions.assertEquals(expectedPath, ((ModifyColumnOp) clause).getColumnPath().getFullPath()); + } else if (clause instanceof ModifyColumnCommentOp) { + Assertions.assertEquals(expectedPath, ((ModifyColumnCommentOp) clause).getColumnPath().getFullPath()); + } else if (clause instanceof DropColumnOp) { + Assertions.assertEquals(expectedPath, ((DropColumnOp) clause).getColumnPath().getFullPath()); + } else if (clause instanceof RenameColumnOp) { + Assertions.assertEquals(expectedPath, ((RenameColumnOp) clause).getColumnPath().getFullPath()); + } + return clause; + } + + private T assertSingleClause(String sql, Class clauseClass) { + Plan plan = parser.parseSingle(sql); + Assertions.assertInstanceOf(AlterTableCommand.class, plan); + List clauses = ((AlterTableCommand) plan).getNereidsOps(); + Assertions.assertEquals(1, clauses.size()); + AlterTableOp clause = clauses.get(0); + Assertions.assertInstanceOf(clauseClass, clause); + return clauseClass.cast(clause); + } +} diff --git a/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/PruneNestedColumnTest.java b/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/PruneNestedColumnTest.java index 000b03ffcd2b5c..ef0200aa78d26f 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/PruneNestedColumnTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/PruneNestedColumnTest.java @@ -862,15 +862,24 @@ public void testDataTypeAccessTree() { Assertions.assertEquals("struct>>>", columnType.toSql()); setAccessPathAndAssertType(slot, ImmutableList.of("s", "city"), "STRUCT"); - setAccessPathAndAssertType(slot, ImmutableList.of("s", "data"), "STRUCT>>>"); - setAccessPathAndAssertType(slot, ImmutableList.of("s", "data", "*"), "STRUCT>>>"); - setAccessPathAndAssertType(slot, ImmutableList.of("s", "data", "*", "KEYS"), "STRUCT>>>"); - setAccessPathAndAssertType(slot, ImmutableList.of("s", "data", "*", "VALUES"), "STRUCT>>>"); - setAccessPathAndAssertType(slot, ImmutableList.of("s", "data", "*", "VALUES", "a"), "STRUCT>>>"); - setAccessPathAndAssertType(slot, ImmutableList.of("s", "data", "*", "VALUES", "b"), "STRUCT>>>"); - setAccessPathAndAssertType(slot, ImmutableList.of("s", "data", "*", "*"), "STRUCT>>>"); - setAccessPathAndAssertType(slot, ImmutableList.of("s", "data", "*", "*", "a"), "STRUCT>>>"); - setAccessPathAndAssertType(slot, ImmutableList.of("s", "data", "*", "*", "b"), "STRUCT>>>"); + setAccessPathAndAssertType(slot, ImmutableList.of("s", "data"), + "STRUCT>>>"); + setAccessPathAndAssertType(slot, ImmutableList.of("s", "data", "*"), + "STRUCT>>>"); + setAccessPathAndAssertType(slot, ImmutableList.of("s", "data", "*", "KEYS"), + "STRUCT>>>"); + setAccessPathAndAssertType(slot, ImmutableList.of("s", "data", "*", "VALUES"), + "STRUCT>>>"); + setAccessPathAndAssertType(slot, ImmutableList.of("s", "data", "*", "VALUES", "a"), + "STRUCT>>>"); + setAccessPathAndAssertType(slot, ImmutableList.of("s", "data", "*", "VALUES", "b"), + "STRUCT>>>"); + setAccessPathAndAssertType(slot, ImmutableList.of("s", "data", "*", "*"), + "STRUCT>>>"); + setAccessPathAndAssertType(slot, ImmutableList.of("s", "data", "*", "*", "a"), + "STRUCT>>>"); + setAccessPathAndAssertType(slot, ImmutableList.of("s", "data", "*", "*", "b"), + "STRUCT>>>"); setAccessPathsAndAssertType(slot, ImmutableList.of( diff --git a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/AlterTableCommandTest.java b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/AlterTableCommandTest.java index 9ca8cf4cc956a2..26898bee8ea0d6 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/AlterTableCommandTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/AlterTableCommandTest.java @@ -17,22 +17,38 @@ package org.apache.doris.nereids.trees.plans.commands; +import org.apache.doris.catalog.DatabaseIf; +import org.apache.doris.catalog.Env; +import org.apache.doris.catalog.TableIf; +import org.apache.doris.common.AnalysisException; +import org.apache.doris.datasource.CatalogIf; +import org.apache.doris.datasource.CatalogMgr; +import org.apache.doris.datasource.iceberg.IcebergExternalTable; import org.apache.doris.info.TableNameInfo; +import org.apache.doris.nereids.parser.NereidsParser; +import org.apache.doris.nereids.trees.plans.Plan; +import org.apache.doris.nereids.trees.plans.commands.info.AddColumnsOp; import org.apache.doris.nereids.trees.plans.commands.info.AddPartitionFieldOp; import org.apache.doris.nereids.trees.plans.commands.info.AlterTableOp; import org.apache.doris.nereids.trees.plans.commands.info.DropPartitionFieldOp; import org.apache.doris.nereids.trees.plans.commands.info.EnableFeatureOp; +import org.apache.doris.nereids.trees.plans.commands.info.ModifyColumnOp; import org.apache.doris.nereids.trees.plans.commands.info.ReplacePartitionFieldOp; import org.junit.jupiter.api.Assertions; import org.junit.jupiter.api.Test; +import org.mockito.MockedStatic; +import org.mockito.Mockito; import java.util.ArrayList; +import java.util.Arrays; import java.util.HashMap; import java.util.List; import java.util.Map; public class AlterTableCommandTest { + private final NereidsParser parser = new NereidsParser(); + @Test void testEnableFeatureOp() { List ops = new ArrayList<>(); @@ -148,4 +164,251 @@ void testReplacePartitionFieldOp() { Assertions.assertEquals(alterTableCommand.toSql(), "ALTER TABLE `db`.`test` REPLACE PARTITION KEY bucket(16, id) WITH truncate(5, code) AS code_trunc"); } + + @Test + void testRejectNestedColumnPathForNonIcebergTable() { + TableIf table = Mockito.mock(TableIf.class); + for (String sql : Arrays.asList( + "ALTER TABLE t ADD COLUMN s.c STRING NULL", + "ALTER TABLE t MODIFY COLUMN s.a BIGINT", + "ALTER TABLE t MODIFY COLUMN s.a COMMENT 'nested comment'", + "ALTER TABLE t DROP COLUMN s.c", + "ALTER TABLE t RENAME COLUMN s.c TO c2")) { + AnalysisException exception = Assertions.assertThrows(AnalysisException.class, + () -> AlterTableCommand.checkColumnOperationsSupported(table, parseAlter(sql).getNereidsOps())); + Assertions.assertTrue(exception.getMessage() + .contains("Nested column path is only supported for Iceberg tables")); + } + } + + @Test + void testAllowNestedColumnPathForIcebergTable() throws AnalysisException { + IcebergExternalTable table = Mockito.mock(IcebergExternalTable.class); + AlterTableCommand.checkColumnOperationsSupported(table, + parseAlter("ALTER TABLE t ADD COLUMN s.c STRING NULL").getNereidsOps()); + } + + @Test + void testRejectRequiredNestedColumnForIcebergTable() { + IcebergExternalTable table = Mockito.mock(IcebergExternalTable.class); + AnalysisException exception = Assertions.assertThrows(AnalysisException.class, + () -> AlterTableCommand.checkColumnOperationsSupported(table, + parseAlter("ALTER TABLE t ADD COLUMN s.required_field INT NOT NULL").getNereidsOps())); + Assertions.assertTrue(exception.getMessage() + .contains("New nested field 's.required_field' must be nullable")); + } + + @Test + void testRejectRollupForIcebergColumnOperations() { + IcebergExternalTable table = Mockito.mock(IcebergExternalTable.class); + for (String sql : Arrays.asList( + "ALTER TABLE t ADD COLUMN c STRING NULL TO r1", + "ALTER TABLE t ADD COLUMN s.c STRING NULL TO r1", + "ALTER TABLE t ADD COLUMN (c1 STRING NULL, c2 INT NULL) IN r1", + "ALTER TABLE t DROP COLUMN c FROM r1", + "ALTER TABLE t DROP COLUMN s.c FROM r1", + "ALTER TABLE t MODIFY COLUMN c STRING FROM r1", + "ALTER TABLE t MODIFY COLUMN s.c STRING FROM r1", + "ALTER TABLE t ORDER BY (c1, c2) FROM r1")) { + AnalysisException exception = Assertions.assertThrows(AnalysisException.class, + () -> AlterTableCommand.checkColumnOperationsSupported(table, parseAlter(sql).getNereidsOps())); + Assertions.assertTrue(exception.getMessage() + .contains("Rollup is not supported for Iceberg column operations")); + } + } + + @Test + void testRejectPropertiesForIcebergColumnOperations() { + IcebergExternalTable table = Mockito.mock(IcebergExternalTable.class); + for (String sql : Arrays.asList( + "ALTER TABLE t ADD COLUMN c STRING NULL PROPERTIES ('k' = 'v')", + "ALTER TABLE t ADD COLUMN s.c STRING NULL PROPERTIES ('k' = 'v')", + "ALTER TABLE t ADD COLUMN (c1 STRING NULL, c2 INT NULL) PROPERTIES ('k' = 'v')", + "ALTER TABLE t DROP COLUMN c PROPERTIES ('k' = 'v')", + "ALTER TABLE t DROP COLUMN s.c PROPERTIES ('k' = 'v')", + "ALTER TABLE t MODIFY COLUMN c STRING PROPERTIES ('k' = 'v')", + "ALTER TABLE t MODIFY COLUMN s.c STRING PROPERTIES ('k' = 'v')", + "ALTER TABLE t ORDER BY (c1, c2) PROPERTIES ('k' = 'v')")) { + AnalysisException exception = Assertions.assertThrows(AnalysisException.class, + () -> AlterTableCommand.checkColumnOperationsSupported(table, parseAlter(sql).getNereidsOps())); + Assertions.assertTrue(exception.getMessage() + .contains("PROPERTIES are not supported for Iceberg column operations")); + } + } + + @Test + void testRejectKeyForIcebergAddAndModify() { + IcebergExternalTable table = Mockito.mock(IcebergExternalTable.class); + for (String sql : Arrays.asList( + "ALTER TABLE t ADD COLUMN c INT KEY NULL", + "ALTER TABLE t ADD COLUMN s.c INT KEY NULL", + "ALTER TABLE t ADD COLUMN (c1 INT KEY NULL, c2 INT NULL)", + "ALTER TABLE t MODIFY COLUMN c BIGINT KEY", + "ALTER TABLE t MODIFY COLUMN s.c BIGINT KEY")) { + AnalysisException exception = Assertions.assertThrows(AnalysisException.class, + () -> AlterTableCommand.checkColumnOperationsSupported(table, parseAlter(sql).getNereidsOps())); + Assertions.assertTrue(exception.getMessage() + .contains("KEY is not supported for Iceberg ADD/MODIFY COLUMN")); + } + } + + @Test + void testRejectGeneratedColumnForIcebergAddAndModify() { + IcebergExternalTable table = Mockito.mock(IcebergExternalTable.class); + for (String sql : Arrays.asList( + "ALTER TABLE t ADD COLUMN c INT AS (id + 1) NULL", + "ALTER TABLE t ADD COLUMN s.c INT AS (id + 1) NULL", + "ALTER TABLE t ADD COLUMN (c1 INT AS (id + 1) NULL, c2 INT NULL)", + "ALTER TABLE t MODIFY COLUMN c BIGINT AS (id + 1)", + "ALTER TABLE t MODIFY COLUMN s.c BIGINT AS (id + 1)")) { + AnalysisException exception = Assertions.assertThrows(AnalysisException.class, + () -> AlterTableCommand.checkColumnOperationsSupported(table, parseAlter(sql).getNereidsOps())); + Assertions.assertTrue(exception.getMessage() + .contains("Generated columns are not supported for Iceberg ADD/MODIFY COLUMN")); + } + } + + @Test + void testRejectUnsupportedDefaultChangesForIcebergTable() throws AnalysisException { + IcebergExternalTable table = Mockito.mock(IcebergExternalTable.class); + for (String sql : Arrays.asList( + "ALTER TABLE t MODIFY COLUMN c BIGINT DEFAULT 7", + "ALTER TABLE t MODIFY COLUMN c BIGINT DEFAULT NULL", + "ALTER TABLE t MODIFY COLUMN ts DATETIME DEFAULT CURRENT_TIMESTAMP " + + "ON UPDATE CURRENT_TIMESTAMP")) { + AnalysisException exception = Assertions.assertThrows(AnalysisException.class, + () -> AlterTableCommand.checkColumnOperationsSupported(table, parseAlter(sql).getNereidsOps())); + Assertions.assertTrue(exception.getMessage() + .contains("Modifying default values is not supported for Iceberg columns")); + } + + org.apache.doris.nereids.exceptions.AnalysisException complexDefaultException = Assertions.assertThrows( + org.apache.doris.nereids.exceptions.AnalysisException.class, + () -> AlterTableCommand.checkColumnOperationsSupported(table, parseAlter( + "ALTER TABLE t MODIFY COLUMN st STRUCT DEFAULT 'x'").getNereidsOps())); + Assertions.assertTrue(complexDefaultException.getMessage() + .contains("Struct type column default value just support null")); + + AnalysisException onUpdateException = Assertions.assertThrows(AnalysisException.class, + () -> AlterTableCommand.checkColumnOperationsSupported(table, parseAlter( + "ALTER TABLE t ADD COLUMN ts DATETIME NULL DEFAULT CURRENT_TIMESTAMP " + + "ON UPDATE CURRENT_TIMESTAMP").getNereidsOps())); + Assertions.assertTrue(onUpdateException.getMessage() + .contains("ON UPDATE is not supported for Iceberg ADD COLUMN")); + + AlterTableCommand.checkColumnOperationsSupported(table, + parseAlter("ALTER TABLE t ADD COLUMN c BIGINT NULL DEFAULT 7").getNereidsOps()); + } + + @Test + void testRejectCompoundIcebergColumnOperations() throws AnalysisException { + IcebergExternalTable table = Mockito.mock(IcebergExternalTable.class); + for (String sql : Arrays.asList( + "ALTER TABLE t ADD COLUMN s.good INT NULL, DROP COLUMN m.value.x", + "ALTER TABLE t MODIFY COLUMN c COMMENT 'new comment', ADD COLUMN d INT NULL", + "ALTER TABLE t ADD COLUMN c INT NULL, DROP COLUMN d", + "ALTER TABLE t RENAME COLUMN c TO c2, RENAME COLUMN d TO d2", + "ALTER TABLE t MODIFY COLUMN c COMMENT 'c', MODIFY COLUMN d COMMENT 'd'", + "ALTER TABLE t ORDER BY (c, d), ORDER BY (d, c)")) { + AnalysisException exception = Assertions.assertThrows(AnalysisException.class, + () -> AlterTableCommand.checkColumnOperationsSupported(table, parseAlter(sql).getNereidsOps())); + Assertions.assertTrue(exception.getMessage() + .contains("Multiple Iceberg ALTER clauses are not supported")); + } + + AlterTableCommand.checkColumnOperationsSupported(table, + parseAlter("ALTER TABLE t ADD COLUMN (c1 INT NULL, c2 BIGINT NULL)").getNereidsOps()); + } + + @Test + void testPreserveEmptyAddColumnsValidationForIcebergTable() throws AnalysisException { + IcebergExternalTable table = Mockito.mock(IcebergExternalTable.class); + AddColumnsOp addColumnsOp = new AddColumnsOp(null, null, new HashMap<>()); + + AlterTableCommand.checkColumnOperationsSupported(table, Arrays.asList(addColumnsOp)); + AnalysisException exception = Assertions.assertThrows(AnalysisException.class, + () -> addColumnsOp.validate(null)); + Assertions.assertTrue(exception.getMessage().contains("Columns is empty in add columns clause")); + } + + @Test + void testModifyColumnTracksExplicitNullability() { + ModifyColumnOp omitted = (ModifyColumnOp) parseAlter( + "ALTER TABLE t MODIFY COLUMN s.a BIGINT").getNereidsOps().get(0); + ModifyColumnOp nullable = (ModifyColumnOp) parseAlter( + "ALTER TABLE t MODIFY COLUMN s.a BIGINT NULL").getNereidsOps().get(0); + ModifyColumnOp notNullable = (ModifyColumnOp) parseAlter( + "ALTER TABLE t MODIFY COLUMN s.a BIGINT NOT NULL").getNereidsOps().get(0); + + Assertions.assertFalse(omitted.getColumnDef() + .translateToCatalogStyleForSchemaChange().isNullableSpecified()); + Assertions.assertTrue(nullable.getColumnDef() + .translateToCatalogStyleForSchemaChange().isNullableSpecified()); + Assertions.assertTrue(notNullable.getColumnDef() + .translateToCatalogStyleForSchemaChange().isNullableSpecified()); + } + + @Test + void testNestedIcebergColumnNamesBypassTopLevelSystemPrefixes() throws Exception { + Env env = Mockito.mock(Env.class); + CatalogMgr catalogMgr = Mockito.mock(CatalogMgr.class); + CatalogIf catalog = Mockito.mock(CatalogIf.class); + DatabaseIf database = Mockito.mock(DatabaseIf.class); + IcebergExternalTable table = Mockito.mock(IcebergExternalTable.class); + Mockito.when(env.getCatalogMgr()).thenReturn(catalogMgr); + Mockito.when(catalogMgr.getCatalogOrDdlException("iceberg")).thenReturn(catalog); + Mockito.when(catalog.getDbOrDdlException("db")).thenReturn(database); + Mockito.when(database.getTableOrDdlException("t")).thenReturn(table); + + try (MockedStatic mockedEnv = Mockito.mockStatic(Env.class)) { + mockedEnv.when(Env::getCurrentEnv).thenReturn(env); + + for (String sql : Arrays.asList( + "ALTER TABLE t ADD COLUMN s.__DORIS_metric INT NULL", + "ALTER TABLE t ADD COLUMN s.__doris_shadow_metric INT NULL", + "ALTER TABLE t MODIFY COLUMN s.__DORIS_metric BIGINT", + "ALTER TABLE t MODIFY COLUMN s.__doris_shadow_metric BIGINT", + "ALTER TABLE t RENAME COLUMN s.__DORIS_metric TO metric", + "ALTER TABLE t RENAME COLUMN s.metric TO __doris_shadow_metric", + "ALTER TABLE t ADD COLUMN s._row_id BIGINT NULL", + "ALTER TABLE t MODIFY COLUMN s._row_id BIGINT", + "ALTER TABLE t RENAME COLUMN s.metric TO _last_updated_sequence_number", + "ALTER TABLE t DROP COLUMN s.__DORIS_metric")) { + validateIcebergAlter(sql, table); + } + + for (String sql : Arrays.asList( + "ALTER TABLE t ADD COLUMN __DORIS_metric INT NULL", + "ALTER TABLE t ADD COLUMN __doris_shadow_metric INT NULL", + "ALTER TABLE t MODIFY COLUMN __DORIS_metric BIGINT")) { + org.apache.doris.nereids.exceptions.AnalysisException exception = Assertions.assertThrows( + org.apache.doris.nereids.exceptions.AnalysisException.class, + () -> validateIcebergAlter(sql, table)); + Assertions.assertTrue(exception.getMessage().contains("Incorrect column name")); + } + + AnalysisException renameException = Assertions.assertThrows(AnalysisException.class, + () -> validateIcebergAlter( + "ALTER TABLE t RENAME COLUMN metric TO __DORIS_metric", table)); + Assertions.assertTrue(renameException.getMessage().contains("Incorrect column name")); + AnalysisException dropException = Assertions.assertThrows(AnalysisException.class, + () -> validateIcebergAlter("ALTER TABLE t DROP COLUMN __DORIS_metric", table)); + Assertions.assertTrue(dropException.getMessage().contains("Do not support drop hidden column")); + } + } + + private void validateIcebergAlter(String sql, IcebergExternalTable table) throws Exception { + AlterTableCommand command = parseAlter(sql); + AlterTableCommand.checkColumnOperationsSupported(table, command.getNereidsOps()); + for (AlterTableOp op : command.getNereidsOps()) { + op.setTableName(new TableNameInfo("iceberg", "db", "t")); + op.validate(null); + } + } + + private AlterTableCommand parseAlter(String sql) { + Plan plan = parser.parseSingle(sql); + Assertions.assertInstanceOf(AlterTableCommand.class, plan); + return (AlterTableCommand) plan; + } } diff --git a/gensrc/thrift/PlanNodes.thrift b/gensrc/thrift/PlanNodes.thrift index 8ae188a73118f0..a76e4543b38f30 100644 --- a/gensrc/thrift/PlanNodes.thrift +++ b/gensrc/thrift/PlanNodes.thrift @@ -111,7 +111,11 @@ enum TFileFormatType { FORMAT_WAL = 15, FORMAT_ARROW = 16, FORMAT_TEXT = 17, - FORMAT_NATIVE = 18 + FORMAT_NATIVE = 18, + // Reserve the master wire IDs so branch-4.1 can explicitly keep these unsupported formats + // off FileScannerV2 without renumbering later TFileFormatType values. + FORMAT_LANCE = 19, + FORMAT_ES_HTTP = 20 } // In previous versions, the data compression format and file format were stored together, as TFileFormatType, diff --git a/regression-test/data/external_table_p0/iceberg/iceberg_schema_change_ddl.out b/regression-test/data/external_table_p0/iceberg/iceberg_schema_change_ddl.out index dab31ada31b925..4565871aa8aa71 100644 --- a/regression-test/data/external_table_p0/iceberg/iceberg_schema_change_ddl.out +++ b/regression-test/data/external_table_p0/iceberg/iceberg_schema_change_ddl.out @@ -149,7 +149,7 @@ phone text Yes true \N User phone number grade double Yes true \N email text Yes true \N address struct Yes true \N -col1 double Yes true \N +col1 double Yes true \N Updated column1 type col2 text Yes true \N User defined column2 -- !after_no_comment -- @@ -329,4 +329,3 @@ department_id bigint Yes true \N 2 Bob 30 9223372036854775806 3 Charlie 22 100 3 Charlie 22 9223372036854775805 - diff --git a/regression-test/data/external_table_p0/iceberg/test_iceberg_multi_spec_filter_aggregate.out b/regression-test/data/external_table_p0/iceberg/test_iceberg_multi_spec_filter_aggregate.out new file mode 100644 index 00000000000000..1875865c3bc201 --- /dev/null +++ b/regression-test/data/external_table_p0/iceberg/test_iceberg_multi_spec_filter_aggregate.out @@ -0,0 +1,35 @@ +-- This file is automatically generated. You should know what you did if you want to edit this +-- !multi_spec_rows -- +1 aa-01 10 +11 bb-11 110 +12 bb-12 120 +13 bb-13 130 +14 bb-14 140 +2 aa-02 20 +3 aa-03 30 +4 aa-04 40 + +-- !multi_spec_static_filter -- +1 aa-01 +12 bb-12 +14 bb-14 +3 aa-03 + +-- !multi_spec_runtime_filter -- +12 bb-12 120 +14 bb-14 140 +2 aa-02 20 +3 aa-03 30 + +-- !multi_spec_grouped -- +aa 4 4 100 +bb 4 4 500 + +-- !multi_spec_distinct -- +8 8 8 + +-- !multi_spec_metadata -- +0 2 +1 2 +2 2 +3 2 diff --git a/regression-test/data/external_table_p0/iceberg/test_iceberg_nested_schema_evolution_ddl.out b/regression-test/data/external_table_p0/iceberg/test_iceberg_nested_schema_evolution_ddl.out new file mode 100644 index 00000000000000..93826ea80b7925 --- /dev/null +++ b/regression-test/data/external_table_p0/iceberg/test_iceberg_nested_schema_evolution_ddl.out @@ -0,0 +1,12 @@ +-- This file is automatically generated. You should know what you did if you want to edit this +-- !desc -- +id int(11) +s struct +arr array> +m map> +arr_scalar array +m_scalar map + +-- !query_rows -- +1 \N 10 \N \N 100 \N \N 1000 \N \N 7 70 +2 first 20 after_a c2 200 202 201 2000 2002 2001 8 80 diff --git a/regression-test/data/external_table_p0/iceberg/test_iceberg_nested_schema_evolution_spark_doris_interop.out b/regression-test/data/external_table_p0/iceberg/test_iceberg_nested_schema_evolution_spark_doris_interop.out new file mode 100644 index 00000000000000..54a02e79582045 --- /dev/null +++ b/regression-test/data/external_table_p0/iceberg/test_iceberg_nested_schema_evolution_spark_doris_interop.out @@ -0,0 +1,27 @@ +-- This file is automatically generated. You should know what you did if you want to edit this +-- !doris_driven_rows -- +1 \N 10 \N doris_before \N \N 100 \N \N \N 1000 \N \N +2 spark_first 20 spark_after_metric spark_after_doris spark_can_write_doris_field 202 200 203 201 2002 2000 2003 2001 + +-- !spark_driven_schema -- +id int(11) +info struct +events array> +attrs map> + +-- !spark_driven_rows_before_write -- +1 \N 10 \N spark_before \N \N 100 \N \N \N 1000 \N \N +2 spark_first_field 20 spark_after_metric_field spark_after spark_new_field 202 200 203 201 2002 2000 2003 2001 + +-- !spark_driven_rows_after_write -- +1 \N 10 \N spark_before \N \N 100 \N \N \N 1000 \N \N +2 spark_first_field 20 spark_after_metric_field spark_after spark_new_field 202 200 203 201 2002 2000 2003 2001 +3 doris_first_field 30 doris_after_metric_field doris_after_spark doris_can_write_spark_field 302 300 303 301 3002 3000 3003 3001 + +-- !required_nested_rows -- +1 10 old-label +2 20 \N + +-- !deep_nested_rows -- +1 last-old middle-old first-old old-note 1.5 12.34 +2 last-new middle-new first-new new-note 2.5 56.78 diff --git a/regression-test/data/external_table_p0/iceberg/test_iceberg_partition_evolution_equality_delete.out b/regression-test/data/external_table_p0/iceberg/test_iceberg_partition_evolution_equality_delete.out new file mode 100644 index 00000000000000..5c4002a33f02ba --- /dev/null +++ b/regression-test/data/external_table_p0/iceberg/test_iceberg_partition_evolution_equality_delete.out @@ -0,0 +1,36 @@ +-- This file is automatically generated. You should know what you did if you want to edit this +-- !current_filter -- +1 +5 +7 + +-- !current_nested -- +7 7000000000 + +-- !base_tag -- +1 +2 + +-- !first_delete_snapshot -- +1 + +-- !added_snapshot -- +1 +5 +6 + +-- !final_tag -- +1 +5 +7 + +-- !scanner_v2 -- +1 +5 +7 + +-- !scanner_v1 -- +1 +5 +7 + diff --git a/regression-test/data/external_table_p0/iceberg/test_iceberg_partition_evolution_filter_refs.out b/regression-test/data/external_table_p0/iceberg/test_iceberg_partition_evolution_filter_refs.out new file mode 100644 index 00000000000000..e4a7f2ca565af5 --- /dev/null +++ b/regression-test/data/external_table_p0/iceberg/test_iceberg_partition_evolution_filter_refs.out @@ -0,0 +1,71 @@ +-- This file is automatically generated. You should know what you did if you want to edit this +-- !current_identity_filter -- +1 +3 +5 +7 + +-- !current_truncate_source_filter -- +1 +3 +5 +7 + +-- !current_temporal_source_filter -- +5 +6 +7 +8 + +-- !current_readded_nested_field -- +7 7000 +8 8000 + +-- !identity_base_snapshot -- +1 A +2 B + +-- !identity_base_tag -- +1 A +2 B + +-- !identity_base_branch -- +1 A +2 B + +-- !identity_added_snapshot -- +1 +3 + +-- !identity_replaced_snapshot -- +1 +3 +5 + +-- !identity_dropped_tag -- +1 +3 +5 +7 + +-- !temporal_year_tag -- +11 +12 + +-- !temporal_current_all_specs -- +11 +12 +13 +14 +15 +16 +17 +18 + +-- !temporal_hour_snapshot -- +17 +18 + +-- !temporal_hour_tag -- +18 + diff --git a/regression-test/data/external_table_p0/iceberg/test_iceberg_partition_evolution_format_scanner.out b/regression-test/data/external_table_p0/iceberg/test_iceberg_partition_evolution_format_scanner.out new file mode 100644 index 00000000000000..c8cd07156d56aa --- /dev/null +++ b/regression-test/data/external_table_p0/iceberg/test_iceberg_partition_evolution_format_scanner.out @@ -0,0 +1,153 @@ +-- This file is automatically generated. You should know what you did if you want to edit this +-- !identity_parquet_v2_current -- +1 \N +2 \N +3 \N +4 \N +5 \N +6 \N +7 7000 +8 8000 + +-- !identity_parquet_v2_base_tag -- +1 A base-a +2 B base-b + +-- !identity_parquet_v2_final_tag -- +1 base-a +3 bucket-a +5 truncate-a +7 drop-a + +-- !temporal_parquet_v2_current -- +11 +12 +13 +14 +15 +16 +17 +18 + +-- !temporal_parquet_v2_final_tag -- +13 +14 +15 +16 +17 +18 + +-- !identity_parquet_v1_current -- +1 \N +2 \N +3 \N +4 \N +5 \N +6 \N +7 7000 +8 8000 + +-- !identity_parquet_v1_base_tag -- +1 A base-a +2 B base-b + +-- !identity_parquet_v1_final_tag -- +1 base-a +3 bucket-a +5 truncate-a +7 drop-a + +-- !temporal_parquet_v1_current -- +11 +12 +13 +14 +15 +16 +17 +18 + +-- !temporal_parquet_v1_final_tag -- +13 +14 +15 +16 +17 +18 + +-- !identity_orc_v2_current -- +1 \N +2 \N +3 \N +4 \N +5 \N +6 \N +7 7000 +8 8000 + +-- !identity_orc_v2_base_tag -- +1 A base-a +2 B base-b + +-- !identity_orc_v2_final_tag -- +1 base-a +3 bucket-a +5 truncate-a +7 drop-a + +-- !temporal_orc_v2_current -- +11 +12 +13 +14 +15 +16 +17 +18 + +-- !temporal_orc_v2_final_tag -- +13 +14 +15 +16 +17 +18 + +-- !identity_orc_v1_current -- +1 \N +2 \N +3 \N +4 \N +5 \N +6 \N +7 7000 +8 8000 + +-- !identity_orc_v1_base_tag -- +1 A base-a +2 B base-b + +-- !identity_orc_v1_final_tag -- +1 base-a +3 bucket-a +5 truncate-a +7 drop-a + +-- !temporal_orc_v1_current -- +11 +12 +13 +14 +15 +16 +17 +18 + +-- !temporal_orc_v1_final_tag -- +13 +14 +15 +16 +17 +18 + diff --git a/regression-test/data/external_table_p0/iceberg/test_iceberg_partition_evolution_position_dv.out b/regression-test/data/external_table_p0/iceberg/test_iceberg_partition_evolution_position_dv.out new file mode 100644 index 00000000000000..bba7dbde155b71 --- /dev/null +++ b/regression-test/data/external_table_p0/iceberg/test_iceberg_partition_evolution_position_dv.out @@ -0,0 +1,201 @@ +-- This file is automatically generated. You should know what you did if you want to edit this +-- !position_parquet_current_filter -- +1 +5 +7 +10 + +-- !position_parquet_current_nested -- +7 7000000000 + +-- !position_parquet_rf_disabled -- +1 +5 +7 +10 + +-- !position_parquet_base_snapshot -- +1 +2 +3 +4 + +-- !position_parquet_base_tag -- +1 +2 + +-- !position_parquet_first_delete_snapshot -- +1 + +-- !position_parquet_added_delete_snapshot -- +1 +5 + +-- !position_parquet_final_tag -- +1 +5 +7 +10 + +-- !position_parquet_scanner_v2 -- +1 +5 +7 +10 + +-- !position_parquet_scanner_v1 -- +1 +5 +7 +10 + +-- !dv_parquet_current_filter -- +1 +5 +7 +10 + +-- !dv_parquet_current_nested -- +7 7000000000 + +-- !dv_parquet_rf_disabled -- +1 +5 +7 +10 + +-- !dv_parquet_base_snapshot -- +1 +2 +3 +4 + +-- !dv_parquet_base_tag -- +1 +2 + +-- !dv_parquet_first_delete_snapshot -- +1 + +-- !dv_parquet_added_delete_snapshot -- +1 +5 + +-- !dv_parquet_final_tag -- +1 +5 +7 +10 + +-- !dv_parquet_scanner_v2 -- +1 +5 +7 +10 + +-- !dv_parquet_scanner_v1 -- +1 +5 +7 +10 + +-- !position_orc_current_filter -- +1 +5 +7 +10 + +-- !position_orc_current_nested -- +7 7000000000 + +-- !position_orc_rf_disabled -- +1 +5 +7 +10 + +-- !position_orc_base_snapshot -- +1 +2 +3 +4 + +-- !position_orc_base_tag -- +1 +2 + +-- !position_orc_first_delete_snapshot -- +1 + +-- !position_orc_added_delete_snapshot -- +1 +5 + +-- !position_orc_final_tag -- +1 +5 +7 +10 + +-- !position_orc_scanner_v2 -- +1 +5 +7 +10 + +-- !position_orc_scanner_v1 -- +1 +5 +7 +10 + +-- !dv_orc_current_filter -- +1 +5 +7 +10 + +-- !dv_orc_current_nested -- +7 7000000000 + +-- !dv_orc_rf_disabled -- +1 +5 +7 +10 + +-- !dv_orc_base_snapshot -- +1 +2 +3 +4 + +-- !dv_orc_base_tag -- +1 +2 + +-- !dv_orc_first_delete_snapshot -- +1 + +-- !dv_orc_added_delete_snapshot -- +1 +5 + +-- !dv_orc_final_tag -- +1 +5 +7 +10 + +-- !dv_orc_scanner_v2 -- +1 +5 +7 +10 + +-- !dv_orc_scanner_v1 -- +1 +5 +7 +10 + diff --git a/regression-test/data/external_table_p0/iceberg/test_iceberg_partition_evolution_runtime_filter.out b/regression-test/data/external_table_p0/iceberg/test_iceberg_partition_evolution_runtime_filter.out new file mode 100644 index 00000000000000..78518f51a42aff --- /dev/null +++ b/regression-test/data/external_table_p0/iceberg/test_iceberg_partition_evolution_runtime_filter.out @@ -0,0 +1,44 @@ +-- This file is automatically generated. You should know what you did if you want to edit this +-- !category_rf_disabled -- +1 +4 +6 +8 + +-- !added_identity_rf_disabled -- +1 +4 +6 +8 + +-- !bucket_source_rf_disabled -- +4 + +-- !temporal_rf_disabled -- +6 +7 +8 +9 + +-- !bucket_source_rf_enabled -- +4 + +-- !temporal_rf_enabled -- +6 +7 +8 +9 + +-- !base_snapshot_rf -- +1 + +-- !added_snapshot_rf -- +1 +4 + +-- !dropped_tag_rf -- +1 +4 +6 +8 + diff --git a/regression-test/data/external_table_p0/iceberg/test_iceberg_schema_evolution_filter_binding.out b/regression-test/data/external_table_p0/iceberg/test_iceberg_schema_evolution_filter_binding.out new file mode 100644 index 00000000000000..6f263263055a86 --- /dev/null +++ b/regression-test/data/external_table_p0/iceberg/test_iceberg_schema_evolution_filter_binding.out @@ -0,0 +1,21 @@ +-- This file is automatically generated. You should know what you did if you want to edit this +-- !schema_filter_warm -- +1 old-a 10 +2 old-b 20 +3 old-c 30 + +-- !schema_filter_static -- +2 old-b 20 +4 new-d 4000000000 + +-- !schema_filter_runtime -- +2 old-b 20 +4 new-d 4000000000 + +-- !schema_filter_aggregate -- +2 5000000020 + +-- !schema_filter_null_semantics -- +2 old-b +3 old-c +4 new-d diff --git a/regression-test/data/external_table_p0/iceberg/write/test_iceberg_failed_write_atomicity_retry.out b/regression-test/data/external_table_p0/iceberg/write/test_iceberg_failed_write_atomicity_retry.out new file mode 100644 index 00000000000000..b6c590e22b7efb --- /dev/null +++ b/regression-test/data/external_table_p0/iceberg/write/test_iceberg_failed_write_atomicity_retry.out @@ -0,0 +1,9 @@ +-- This file is automatically generated. You should know what you did if you want to edit this +-- !failed_write_state -- +1 committed + +-- !failed_write_retry -- +1 committed 1 +2 candidate-0 1 +3 candidate-1 1 +4 candidate-2 1 diff --git a/regression-test/data/external_table_p0/iceberg/write/test_iceberg_write_branch_dml_boundary.out b/regression-test/data/external_table_p0/iceberg/write/test_iceberg_write_branch_dml_boundary.out new file mode 100644 index 00000000000000..5795463b78bf0c --- /dev/null +++ b/regression-test/data/external_table_p0/iceberg/write/test_iceberg_write_branch_dml_boundary.out @@ -0,0 +1,12 @@ +-- This file is automatically generated. You should know what you did if you want to edit this +-- !branch_write -- +1 A main +2 B branch-insert +3 C branch-overwrite +-- !main_after_branch_write -- +1 A main + +-- !branch_after_rejected_dml -- +1 A main +2 B branch-insert +3 C branch-overwrite diff --git a/regression-test/data/external_table_p0/iceberg/write/test_iceberg_write_complex_evolution.out b/regression-test/data/external_table_p0/iceberg/write/test_iceberg_write_complex_evolution.out new file mode 100644 index 00000000000000..76bede82d3f5fb --- /dev/null +++ b/regression-test/data/external_table_p0/iceberg/write/test_iceberg_write_complex_evolution.out @@ -0,0 +1,30 @@ +-- This file is automatically generated. You should know what you did if you want to edit this +-- !complex_current -- +1 A [1, null, 3] {"x":10, "null-value":null} {"metric":10, "label":"old-a", "nested":{"count":1, "comment":null, "score":null}, "tags":null, "attributes":null} +2 N \N {"x":null} {"metric":20, "label":null, "nested":{"count":null, "comment":"old-null", "score":null}, "tags":null, "attributes":null} +3 B [] {} \N +4 A1 [4000000000, null] {"large":5000000000, "null-value":null} {"metric":6000000000, "label":"new-a", "nested":{"count":7000000000, "comment":"nested-new", "score":7.5}, "tags":["x", null, "z"], "attributes":{"a":8000000000, "b":null}} +5 N2 [null] \N {"metric":50, "label":null, "nested":{"count":5, "comment":null, "score":null}, "tags":null, "attributes":{"null-value":null}} + +-- !complex_children -- +1 10 1 \N \N \N +2 20 \N \N \N \N +3 \N \N \N \N \N +4 6000000000 7000000000 7.5 ["x", null, "z"] {"a":8000000000, "b":null} +5 50 5 \N \N {"null-value":null} + +-- !complex_nulls -- +1 +2 +3 +5 + +-- !complex_partition_specs -- +0 3 +2 2 + +-- !complex_base_tag -- +1 [1, null, 3] {"x":10, "null-value":null} 10 old-a 1 \N +2 \N {"x":null} 20 \N \N old-null +3 [] {} \N \N \N \N + diff --git a/regression-test/data/external_table_p0/iceberg/write/test_iceberg_write_concurrent_merge_invariants.out b/regression-test/data/external_table_p0/iceberg/write/test_iceberg_write_concurrent_merge_invariants.out new file mode 100644 index 00000000000000..2ab2c582fe3762 --- /dev/null +++ b/regression-test/data/external_table_p0/iceberg/write/test_iceberg_write_concurrent_merge_invariants.out @@ -0,0 +1,4 @@ +-- This file is automatically generated. You should know what you did if you want to edit this +-- !concurrent_append_counts -- +append-one 128 128 +append-two 128 128 diff --git a/regression-test/data/external_table_p0/iceberg/write/test_iceberg_write_ctas_format_boundary.out b/regression-test/data/external_table_p0/iceberg/write/test_iceberg_write_ctas_format_boundary.out new file mode 100644 index 00000000000000..4a418a9b8cd01e --- /dev/null +++ b/regression-test/data/external_table_p0/iceberg/write/test_iceberg_write_ctas_format_boundary.out @@ -0,0 +1,16 @@ +-- This file is automatically generated. You should know what you did if you want to edit this +-- !ctas_complex_rows -- +1 A ["x", null] {"k":"v"} {"score":10, "note":"one"} +2 \N [] {"null-value":null} {"score":null, "note":"two"} +3 A ["😀"] {} {"score":30, "note":null} + +-- !ctas_complex_files -- +orc 3 + +-- !ctas_complex_partitions -- +0 3 3 + +-- !ctas_complex_physical_partitions -- +\N 0 1 +A 0 1 +A 3 1 diff --git a/regression-test/data/external_table_p0/iceberg/write/test_iceberg_write_dml_modes_evolution.out b/regression-test/data/external_table_p0/iceberg/write/test_iceberg_write_dml_modes_evolution.out new file mode 100644 index 00000000000000..98c60ec0eb83ee --- /dev/null +++ b/regression-test/data/external_table_p0/iceberg/write/test_iceberg_write_dml_modes_evolution.out @@ -0,0 +1,32 @@ +-- This file is automatically generated. You should know what you did if you want to edit this +-- !mor_current -- +1 A-updated alpha 2026-01-01T01:00 110 updated +2 B-merged beta-merged 2026-04-02T02:00 220 merged +4 A-updated delta 2026-01-04T04:00 140 updated +7 C golf 2026-03-01T07:00 70 \N +8 D hotel 2026-05-01T08:00 80 inserted + +-- !mor_base_tag -- +1 A alpha 2026-01-01T01:00 10 +2 B beta 2026-01-02T02:00 20 +3 \N null-key \N 30 +4 A delta 2026-01-04T04:00 40 + +-- !mor_before_dml_tag -- +1 A alpha 2026-01-01T01:00 10 \N +2 B beta 2026-01-02T02:00 20 \N +3 \N null-key \N 30 \N +4 A delta 2026-01-04T04:00 40 \N +5 B echo 2026-02-01T05:00 50 new-spec +6 \N foxtrot \N 60 new-null +7 C golf 2026-03-01T07:00 70 \N + +-- !mor_delete_files -- +0 4 4 +2 2 2 + +-- !cow_after_rejections -- +1 A alpha 2026-01-01T01:00 10 base +2 \N null-key \N 20 null-partition +3 B beta 2026-02-01T03:00 30 new-spec + diff --git a/regression-test/data/external_table_p0/iceberg/write/test_iceberg_write_evolution_refs.out b/regression-test/data/external_table_p0/iceberg/write/test_iceberg_write_evolution_refs.out new file mode 100644 index 00000000000000..f7380c14c7ded5 --- /dev/null +++ b/regression-test/data/external_table_p0/iceberg/write/test_iceberg_write_evolution_refs.out @@ -0,0 +1,75 @@ +-- This file is automatically generated. You should know what you did if you want to edit this +-- !current_rows -- +1 CN alpha 2026-01-01T08:00 10.10 10 \N \N +2 US beta 2026-01-02T09:00 20.20 20 \N \N +3 \N null-key \N 30.30 30 \N \N +4 CN-east gamma 2026-02-01T10:00 40.40 4000000000 after-evolution new-spec +5 DE-west delta 2026-03-02T11:00 50.50 50 \N \N +6 \N epsilon \N 60.60 60 null-partition null-zone + +-- !cross_spec_zone_filter -- +1 +4 + +-- !cross_spec_time_filter -- +3 +4 +5 +6 + +-- !partition_specs -- +0 3 +4 3 + +-- !base_snapshot -- +1 CN +2 US +3 \N + +-- !base_tag -- +1 CN +2 US +3 \N + +-- !evolved_tag -- +1 CN \N +2 US \N +3 \N \N +4 CN-east new-spec +5 DE-west \N +6 \N null-zone + +-- !branch_after_insert -- +1 CN \N +2 US \N +3 \N \N +7 JP-east branch-insert +8 FR-west branch-overwrite-seed + +-- !main_unchanged_after_branch_insert -- +1 +2 +3 +4 +5 +6 + +-- !branch_after_overwrite -- +1 CN \N +2 US \N +3 \N \N +7 JP-east branch-insert +8 FR-west branch-overwrite + +-- !base_tag_after_branch_overwrite -- +1 CN +2 US +3 \N + +-- !main_after_branch_overwrite -- +1 CN \N +2 US \N +3 \N \N +4 CN-east new-spec +5 DE-west \N +6 \N null-zone diff --git a/regression-test/data/external_table_p0/iceberg/write/test_iceberg_write_merge_duplicate_source_negative.out b/regression-test/data/external_table_p0/iceberg/write/test_iceberg_write_merge_duplicate_source_negative.out new file mode 100644 index 00000000000000..3f8c510addc24d --- /dev/null +++ b/regression-test/data/external_table_p0/iceberg/write/test_iceberg_write_merge_duplicate_source_negative.out @@ -0,0 +1,3 @@ +-- This file is automatically generated. You should know what you did if you want to edit this +-- !duplicate_source_atomic_state -- +1 A committed diff --git a/regression-test/data/external_table_p0/iceberg/write/test_iceberg_write_merge_semantics.out b/regression-test/data/external_table_p0/iceberg/write/test_iceberg_write_merge_semantics.out new file mode 100644 index 00000000000000..2d9051a0ac1cd0 --- /dev/null +++ b/regression-test/data/external_table_p0/iceberg/write/test_iceberg_write_merge_semantics.out @@ -0,0 +1,13 @@ +-- This file is automatically generated. You should know what you did if you want to edit this +-- !merge_conditional_clauses -- +1 A2 bucket-a2 delta new-1 updated +2 \N bucket-b beta old-2 active +4 \N bucket-d echo new-4 insert-1 +5 E bucket-e foxtrot new-5 insert-2 + +-- !merge_string_partition_metadata -- +0 6 6 + +-- !merge_null_keys -- +\N \N \N \N source-null-safe null-safe-update +\N \N \N \N source-ordinary ordinary-insert diff --git a/regression-test/data/external_table_p0/iceberg/write/test_iceberg_write_merge_truncate_negative.out b/regression-test/data/external_table_p0/iceberg/write/test_iceberg_write_merge_truncate_negative.out new file mode 100644 index 00000000000000..6a9d3165c3474d --- /dev/null +++ b/regression-test/data/external_table_p0/iceberg/write/test_iceberg_write_merge_truncate_negative.out @@ -0,0 +1,9 @@ +-- This file is automatically generated. You should know what you did if you want to edit this +-- !merge_truncate_after_fix -- +1 beta after +2 gamma inserted + +-- !merge_truncate_physical_partitions -- +616C +6265 +6761 diff --git a/regression-test/data/external_table_p0/iceberg/write/test_iceberg_write_nullability_atomicity.out b/regression-test/data/external_table_p0/iceberg/write/test_iceberg_write_nullability_atomicity.out new file mode 100644 index 00000000000000..19fc09fe26cf1c --- /dev/null +++ b/regression-test/data/external_table_p0/iceberg/write/test_iceberg_write_nullability_atomicity.out @@ -0,0 +1,7 @@ +-- This file is automatically generated. You should know what you did if you want to edit this +-- !required_after_retry -- +1 committed \N +2 valid-select \N +4 valid-after-invalid value +5 valid-values-retry \N + diff --git a/regression-test/data/external_table_p0/iceberg/write/test_iceberg_write_nullable_truncate_negative.out b/regression-test/data/external_table_p0/iceberg/write/test_iceberg_write_nullable_truncate_negative.out new file mode 100644 index 00000000000000..bb09eac2571535 --- /dev/null +++ b/regression-test/data/external_table_p0/iceberg/write/test_iceberg_write_nullable_truncate_negative.out @@ -0,0 +1,13 @@ +-- This file is automatically generated. You should know what you did if you want to edit this +-- !nullable_truncate_rows -- +1 CN +2 \N +3 US-west +4 \N + +-- !nullable_truncate_physical_partitions -- +0 \N \N +0 434E \N +1 \N \N +1 55532D65617374 5553 +1 55532D77657374 5553 diff --git a/regression-test/data/external_table_p0/iceberg/write/test_iceberg_write_order_distribution_properties.out b/regression-test/data/external_table_p0/iceberg/write/test_iceberg_write_order_distribution_properties.out new file mode 100644 index 00000000000000..eb0ba541b0e826 --- /dev/null +++ b/regression-test/data/external_table_p0/iceberg/write/test_iceberg_write_order_distribution_properties.out @@ -0,0 +1,14 @@ +-- This file is automatically generated. You should know what you did if you want to edit this +-- !ordered_evolution_changed_rows -- +1 R-updated payload-1-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 10001 updated +10001 R-new merge-insert 10001 inserted +2 \N merge-update 20002 merged +7 R-updated payload-7-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 10007 updated + +-- !ordered_evolution_files -- +parquet 10007 + +-- !distribution_mode_counts -- +hash 512 +none 512 +range 512 diff --git a/regression-test/data/external_table_p0/iceberg/write/test_iceberg_write_overwrite_atomicity.out b/regression-test/data/external_table_p0/iceberg/write/test_iceberg_write_overwrite_atomicity.out new file mode 100644 index 00000000000000..7cfaa975e3e384 --- /dev/null +++ b/regression-test/data/external_table_p0/iceberg/write/test_iceberg_write_overwrite_atomicity.out @@ -0,0 +1,23 @@ +-- This file is automatically generated. You should know what you did if you want to edit this +-- !overwrite_failure_state -- +1 A committed-a +2 B committed-b + +-- !branch_overwrite_failure_state -- +1 A committed-a +2 B committed-b + +-- !main_after_branch_overwrite_failure -- +1 A committed-a +2 B committed-b + +-- !overwrite_retry -- +10 A candidate-0 1 +11 C candidate-1 1 +12 A candidate-2 1 +13 C candidate-3 1 +14 A candidate-4 1 +15 C candidate-5 1 +16 A candidate-6 1 +17 C candidate-7 1 +2 B committed-b 1 diff --git a/regression-test/data/external_table_p0/iceberg/write/test_iceberg_write_overwrite_delete_files.out b/regression-test/data/external_table_p0/iceberg/write/test_iceberg_write_overwrite_delete_files.out new file mode 100644 index 00000000000000..8ab5949d05b624 --- /dev/null +++ b/regression-test/data/external_table_p0/iceberg/write/test_iceberg_write_overwrite_delete_files.out @@ -0,0 +1,40 @@ +-- This file is automatically generated. You should know what you did if you want to edit this +-- !before_overwrite_rows -- +1 A alpha keep-a +3 C gamma-new moved-to-c +5 \N null-key keep-null +6 B echo merge-insert-b + +-- !before_overwrite_delete_files -- +0 3 + +-- !after_overwrite_rows -- +10 A alpha replacement-a +11 B echo replacement-b +3 C gamma-new moved-to-c +5 \N null-key keep-null + +-- !after_overwrite_delete_files -- +0 1 + +-- !before_row_dml_tag -- +1 A alpha keep-a +2 A beta delete-a +3 B gamma move-b-to-c +4 B delta merge-delete-b +5 \N null-key keep-null + +-- !evolved_overwrite_rows -- +10 A alpha replacement-a +11 B echo replacement-b +13 \N null-new new-spec-null +14 A alpha-new new-spec-replacement-a +3 C gamma-new moved-to-c +5 \N null-key keep-null + +-- !evolved_overwrite_specs -- +0 5 5 +2 2 2 + +-- !evolved_overwrite_delete_files -- +0 1 diff --git a/regression-test/data/external_table_p0/iceberg/write/test_iceberg_write_overwrite_evolution.out b/regression-test/data/external_table_p0/iceberg/write/test_iceberg_write_overwrite_evolution.out new file mode 100644 index 00000000000000..a5095dbb0ef041 --- /dev/null +++ b/regression-test/data/external_table_p0/iceberg/write/test_iceberg_write_overwrite_evolution.out @@ -0,0 +1,45 @@ +-- This file is automatically generated. You should know what you did if you want to edit this +-- !overwrite_current -- +1 A alpha 2026-01-01T01:10 old-hour-1 +2 A beta 2026-01-01T02:20 old-hour-2 +3 \N null-region \N old-null +4 B delta 2026-01-02T01:30 old-other-day +6 \N null-new \N new-spec-null +7 C charlie 2026-02-01T03:00 new-spec-other-month +8 A alpha-new 2026-01-01T01:40 overwrite-current-spec + +-- !overwrite_specs -- +0 3 4 +3 3 3 + +-- !overwrite_base_tag -- +1 A alpha 2026-01-01T01:10 old-hour-1 +2 A beta 2026-01-01T02:20 old-hour-2 +3 \N null-region \N old-null +4 B delta 2026-01-02T01:30 old-other-day +-- !overwrite_audit_branch -- +1 A alpha 2026-01-01T01:10 old-hour-1 +2 A beta 2026-01-01T02:20 old-hour-2 +3 \N null-region \N old-null +4 B delta 2026-01-02T01:30 old-other-day + +-- !overwrite_after_drop_identity -- +1 A alpha 2026-01-01T01:10 old-hour-1 +2 A beta 2026-01-01T02:20 old-hour-2 +3 \N null-region \N old-null +4 B delta 2026-01-02T01:30 old-other-day +6 \N null-new \N new-spec-null +7 C charlie 2026-02-01T03:00 new-spec-other-month +8 A alpha-new 2026-01-01T01:40 overwrite-current-spec +9 \N null-new \N overwrite-null-current-spec + +-- !overwrite_after_drop_identity_specs -- +0 3 4 +3 3 3 +5 1 1 + +-- !overwrite_base_tag_after_second_evolution -- +1 A alpha 2026-01-01T01:10 old-hour-1 +2 A beta 2026-01-01T02:20 old-hour-2 +3 \N null-region \N old-null +4 B delta 2026-01-02T01:30 old-other-day diff --git a/regression-test/data/external_table_p0/iceberg/write/test_iceberg_write_partition_types_null.out b/regression-test/data/external_table_p0/iceberg/write/test_iceberg_write_partition_types_null.out new file mode 100644 index 00000000000000..552e5319a9d9b7 --- /dev/null +++ b/regression-test/data/external_table_p0/iceberg/write/test_iceberg_write_partition_types_null.out @@ -0,0 +1,64 @@ +-- This file is automatically generated. You should know what you did if you want to edit this +-- !string_rows -- +1 alpha bucket-a alpha a +2 alphabet bucket-b alphabet ab +3 bucket-empty empty +4 中文 bucket-unicode 中文 unicode +5 \N bucket-null-identity null-identity null-string + +-- !string_null_filter -- +5 + +-- !string_cross_spec_filter -- +1 +2 +5 +7 + +-- !string_cross_spec_bucket_filter -- +1 +6 + +-- !string_partition_specs -- +0 5 +1 2 + +-- !numeric_rows -- +1 1 101 11.11 true positive +2 -1 -101 -11.11 false negative +3 0 0 0.00 \N zero-null-bool +4 \N \N \N \N all-null + +-- !numeric_null_filter -- +3 +4 + +-- !numeric_partitions -- +0 4 + +-- !numeric_physical_partitions -- +\N \N \N \N \N \N 1 +0 4 0 7 0.00 \N 1 +0 4 100 1 11.10 true 1 +0 5 -200 4 -11.20 false 1 + +-- !temporal_rows -- +1 1969-12-31 1969-12-31 1969-12-31 1969-12-31T23:59:59 1969-12-31T23:59:59 1969-12-31T23:59:59 before-epoch +2 1970-01-01 1970-01-01 1970-01-01 1970-01-01T00:00 1970-01-01T00:00 1970-01-01T00:00 epoch +3 2024-02-29 2024-02-29 2024-02-29 2024-02-29T12:34:56 2024-02-29T12:34:56 2024-02-29T12:34:56 leap-day +4 \N \N \N \N \N \N all-null + +-- !temporal_filters -- +1 +3 +4 + +-- !temporal_partitions -- +0 4 + +-- !temporal_physical_partitions -- +\N \N \N \N \N \N 1 +0 0 0 7 1970-01-01 0 1 +4 0 0 4 1970-01-01 0 1 +6 54 649 4 2024-02-29 474780 1 + diff --git a/regression-test/data/external_table_p0/iceberg/write/test_iceberg_write_source_models.out b/regression-test/data/external_table_p0/iceberg/write/test_iceberg_write_source_models.out new file mode 100644 index 00000000000000..1583589eb295bd --- /dev/null +++ b/regression-test/data/external_table_p0/iceberg/write/test_iceberg_write_source_models.out @@ -0,0 +1,26 @@ +-- This file is automatically generated. You should know what you did if you want to edit this +-- !internal_model_oracle -- +aggregate 30 E 303 +aggregate 31 F 310 +duplicate 1 A 10 +duplicate 1 A 11 +duplicate 2 \N 20 +unique_mor 20 C 201 +unique_mor 21 D 210 +unique_mow 10 A 101 +unique_mow 11 \N 110 + +-- !source_model_sink -- +aggregate 30 E 303 +aggregate 31 F 310 +duplicate 1 A 10 +duplicate 1 A 11 +duplicate 2 \N 20 +unique_mor 20 C 201 +unique_mor 21 D 210 +unique_mow 10 A 101 +unique_mow 11 \N 110 + +-- !source_model_partition_stats -- +0 9 + diff --git a/regression-test/data/external_table_p0/iceberg/write/test_iceberg_write_string_transform_metadata.out b/regression-test/data/external_table_p0/iceberg/write/test_iceberg_write_string_transform_metadata.out new file mode 100644 index 00000000000000..a77b5504c2d1f8 --- /dev/null +++ b/regression-test/data/external_table_p0/iceberg/write/test_iceberg_write_string_transform_metadata.out @@ -0,0 +1,40 @@ +-- This file is automatically generated. You should know what you did if you want to edit this +-- !string_transform_rows -- +1 6173636969 6275636B65742D61 616C706861626574 ascii +2 E4B8ADE69687 E6A1B62DE4B8ADE69687 E4B8ADE69687E794B2 cjk +3 656D6F6A69 F09F98802D6275636B6574 F09F9880E794B2E4B999 emoji +4 65CC81 636F6D62696E696E672D6275636B6574 65CC8178 combining +5 empty +6 \N \N 6E756C6C2D6275636B6574 nullable-bucket + +-- !string_transform_physical_partitions -- +\N \N 6E75 1 + 0 1 +ascii 0 616C 1 +emoji 5 F09F9880E794B2 1 +é 0 65CC81 1 +中文 1 E4B8ADE69687 1 + +-- !string_transform_evolved_specs -- +2 6 6 +4 2 2 + +-- !string_transform_evolved_physical_partitions -- +2 \N \N 6E75 \N \N 1 +2 0 \N \N 1 +2 6173636969 0 616C \N \N 1 +2 656D6F6A69 5 F09F9880E794B2 \N \N 1 +2 65CC81 0 65CC81 \N \N 1 +2 E4B8ADE69687 1 E4B8ADE69687 \N \N 1 +4 \N \N \N \N F09F9880E794B2E4B999 1 +4 6E6577 \N \N 6 E4B8ADE69687E794B2 1 + +-- !string_transform_evolved_rows -- +1 6173636969 6275636B65742D61 616C706861626574 ascii +2 E4B8ADE69687 E6A1B62DE4B8ADE69687 E4B8ADE69687E794B2 cjk +3 656D6F6A69 F09F98802D6275636B6574 F09F9880E794B2E4B999 emoji +4 65CC81 636F6D62696E696E672D6275636B6574 65CC8178 combining +5 empty +6 \N \N 6E756C6C2D6275636B6574 nullable-bucket +7 6E6577 6275636B65742D6E6577 E4B8ADE69687E794B2E4B999 new-cjk +8 \N \N F09F9880E794B2E4B999E4B899 new-null-bucket diff --git a/regression-test/data/external_table_p0/paimon/test_paimon_partition_mutation_atomicity.out b/regression-test/data/external_table_p0/paimon/test_paimon_partition_mutation_atomicity.out new file mode 100644 index 00000000000000..e1fcf8c5234718 --- /dev/null +++ b/regression-test/data/external_table_p0/paimon/test_paimon_partition_mutation_atomicity.out @@ -0,0 +1,14 @@ +-- This file is automatically generated. You should know what you did if you want to edit this +-- !current_partition_filter -- +1 +2 +4 + +-- !current_added_payload -- +4 added + +-- !base_tag_partition_filter -- +1 A 1 base-a1 +2 A 2 base-a2 +3 B 1 base-b1 + diff --git a/regression-test/data/external_table_p0/paimon/test_paimon_partition_pk_delete_refs.out b/regression-test/data/external_table_p0/paimon/test_paimon_partition_pk_delete_refs.out new file mode 100644 index 00000000000000..0d8ef36d6335a9 --- /dev/null +++ b/regression-test/data/external_table_p0/paimon/test_paimon_partition_pk_delete_refs.out @@ -0,0 +1,109 @@ +-- This file is automatically generated. You should know what you did if you want to edit this +-- !parquet_current_partition_filter -- +1 alpha-updated +5 epsilon + +-- !parquet_current_promoted_and_readded -- +5 5000000000 5000 + +-- !parquet_current_evolved_fields -- +1 updated-1 extra-1 \N +5 insert-5 extra-5 5000 + +-- !parquet_pre_dv_old_row_null -- +1 updated-1 extra-1 \N + +-- !parquet_rf_disabled -- +1 alpha-updated +5 epsilon + +-- !parquet_base_snapshot -- +1 alpha old-note-1 base-1 +2 beta old-note-2 base-delete + +-- !parquet_base_tag -- +1 alpha old-note-1 base-1 +2 beta old-note-2 base-delete + +-- !parquet_first_delete_snapshot -- +1 alpha-updated new-note-1 updated-1 extra-1 +4 delta delete-later insert-4 extra-4 + +-- !parquet_final_tag -- +1 alpha-updated \N updated-1 extra-1 +5 epsilon 5000 insert-5 extra-5 + +-- !parquet_jni_current_dv -- +1 p1 alpha-updated \N updated-1 extra-1 +5 p1 epsilon 5000 insert-5 extra-5 +3 p2 gamma \N base-p2 \N + +-- !parquet_jni_historical -- +1 p1 alpha old-note-1 base-1 +2 p1 beta old-note-2 base-delete +3 p2 gamma old-note-3 base-p2 + +-- !parquet_native_current_dv -- +1 p1 alpha-updated \N updated-1 extra-1 +5 p1 epsilon 5000 insert-5 extra-5 +3 p2 gamma \N base-p2 \N + +-- !parquet_native_historical -- +1 p1 alpha old-note-1 base-1 +2 p1 beta old-note-2 base-delete +3 p2 gamma old-note-3 base-p2 + +-- !orc_current_partition_filter -- +1 alpha-updated +5 epsilon + +-- !orc_current_promoted_and_readded -- +5 5000000000 5000 + +-- !orc_current_evolved_fields -- +1 updated-1 extra-1 \N +5 insert-5 extra-5 5000 + +-- !orc_pre_dv_old_row_null -- +1 updated-1 extra-1 \N + +-- !orc_rf_disabled -- +1 alpha-updated +5 epsilon + +-- !orc_base_snapshot -- +1 alpha old-note-1 base-1 +2 beta old-note-2 base-delete + +-- !orc_base_tag -- +1 alpha old-note-1 base-1 +2 beta old-note-2 base-delete + +-- !orc_first_delete_snapshot -- +1 alpha-updated new-note-1 updated-1 extra-1 +4 delta delete-later insert-4 extra-4 + +-- !orc_final_tag -- +1 alpha-updated \N updated-1 extra-1 +5 epsilon 5000 insert-5 extra-5 + +-- !orc_jni_current_dv -- +1 p1 alpha-updated \N updated-1 extra-1 +5 p1 epsilon 5000 insert-5 extra-5 +3 p2 gamma \N base-p2 \N + +-- !orc_jni_historical -- +1 p1 alpha old-note-1 base-1 +2 p1 beta old-note-2 base-delete +3 p2 gamma old-note-3 base-p2 + +-- !orc_native_current_dv -- +1 p1 alpha-updated \N updated-1 extra-1 +5 p1 epsilon 5000 insert-5 extra-5 +3 p2 gamma \N base-p2 \N + +-- !orc_native_historical -- +1 p1 alpha old-note-1 base-1 +2 p1 beta old-note-2 base-delete +3 p2 gamma old-note-3 base-p2 + diff --git a/regression-test/data/external_table_p0/paimon/test_paimon_partition_schema_filter_refs.out b/regression-test/data/external_table_p0/paimon/test_paimon_partition_schema_filter_refs.out new file mode 100644 index 00000000000000..ad3c6a41095abd --- /dev/null +++ b/regression-test/data/external_table_p0/paimon/test_paimon_partition_schema_filter_refs.out @@ -0,0 +1,145 @@ +-- This file is automatically generated. You should know what you did if you want to edit this +-- !parquet_current_partition_filter -- +1 +3 +5 +7 + +-- !parquet_current_promoted_metric -- +5 5000000000 + +-- !parquet_current_readded_payload -- +7 7000 + +-- !parquet_current_all_complex_children -- +1 base-1 \N 100 \N 1000 \N +3 add-3 \N 300 301 3000 3001 +5 rename-5 \N 500 501 5000 5001 +7 readd-7 7000 700 701 7000 7001 + +-- !parquet_rf_disabled -- +1 +3 +5 +7 + +-- !parquet_base_snapshot -- +1 p1 base-1 +2 p2 base-2 + +-- !parquet_base_tag -- +1 p1 base-1 +2 p2 base-2 + +-- !parquet_base_branch -- +1 p1 base-1 +2 p2 base-2 + +-- !parquet_added_snapshot_complex -- +1 base-1 100 \N 1000 \N +3 add-3 300 301 3000 3001 + +-- !parquet_final_tag -- +1 +3 +5 +7 + +-- !parquet_base_snapshot_complex -- +1 100 1000 +2 200 2000 + +-- !parquet_jni_current_complex -- +1 base-1 \N 100 1000 +3 add-3 \N 300 3000 +4 add-4 \N 400 4000 +5 rename-5 \N 500 5000 +7 readd-7 7000 700 7000 + +-- !parquet_jni_historical_complex -- +1 base-1 100 1000 +2 base-2 200 2000 + +-- !parquet_native_current_complex -- +1 base-1 \N 100 1000 +3 add-3 \N 300 3000 +4 add-4 \N 400 4000 +5 rename-5 \N 500 5000 +7 readd-7 7000 700 7000 + +-- !parquet_native_historical_complex -- +1 base-1 100 1000 +2 base-2 200 2000 + +-- !orc_current_partition_filter -- +1 +3 +5 +7 + +-- !orc_current_promoted_metric -- +5 5000000000 + +-- !orc_current_readded_payload -- +7 7000 + +-- !orc_current_all_complex_children -- +1 base-1 \N 100 \N 1000 \N +3 add-3 \N 300 301 3000 3001 +5 rename-5 \N 500 501 5000 5001 +7 readd-7 7000 700 701 7000 7001 + +-- !orc_rf_disabled -- +1 +3 +5 +7 + +-- !orc_base_snapshot -- +1 p1 base-1 +2 p2 base-2 + +-- !orc_base_tag -- +1 p1 base-1 +2 p2 base-2 + +-- !orc_base_branch -- +1 p1 base-1 +2 p2 base-2 + +-- !orc_added_snapshot_complex -- +1 base-1 100 \N 1000 \N +3 add-3 300 301 3000 3001 + +-- !orc_final_tag -- +1 +3 +5 +7 + +-- !orc_base_snapshot_complex -- +1 100 1000 +2 200 2000 + +-- !orc_jni_current_complex -- +1 base-1 \N 100 1000 +3 add-3 \N 300 3000 +4 add-4 \N 400 4000 +5 rename-5 \N 500 5000 +7 readd-7 7000 700 7000 + +-- !orc_jni_historical_complex -- +1 base-1 100 1000 +2 base-2 200 2000 + +-- !orc_native_current_complex -- +1 base-1 \N 100 1000 +3 add-3 \N 300 3000 +4 add-4 \N 400 4000 +5 rename-5 \N 500 5000 +7 readd-7 7000 700 7000 + +-- !orc_native_historical_complex -- +1 base-1 100 1000 +2 base-2 200 2000 + diff --git a/regression-test/suites/audit/test_external_catalog_credential_masking.groovy b/regression-test/suites/audit/test_external_catalog_credential_masking.groovy new file mode 100644 index 00000000000000..b7b1bd4aebc530 --- /dev/null +++ b/regression-test/suites/audit/test_external_catalog_credential_masking.groovy @@ -0,0 +1,99 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +suite("test_external_catalog_credential_masking", "p0,nonConcurrent") { + String credentialCatalog = "audit_iceberg_credential_masking" + String tokenCatalog = "audit_iceberg_token_masking" + // Low-entropy placeholders keep this negative masking test distinct from real credentials. + String accessKey = "aaaaaaaaaaaaaaaaaaaa" + String secretKey = "bbbbbbbbbbbbbbbbbbbb" + String oauthCredential = "cccccccccccccccccccc" + String oauthToken = "dddddddddddddddddddd" + + def waitForAuditStatement = { String catalogName -> + String query = """ + select stmt + from __internal_schema.audit_log + where stmt_type = 'OTHER' + and lower(stmt) like '%create catalog ${catalogName}%' + and instr(stmt, '*XXX') > 0 + order by time asc + limit 1 + """ + int retry = 60 + def rows = sql query + while (rows.isEmpty()) { + if (retry-- < 0) { + throw new RuntimeException("audit statement for ${catalogName} was not found") + } + sleep(1000) + sql """call flush_audit_log()""" + rows = sql query + } + return rows[0][0].toString() + } + + setGlobalVarTemporary([enable_audit_plugin: true], { + try { + sql """drop catalog if exists ${credentialCatalog}""" + sql """drop catalog if exists ${tokenCatalog}""" + sql """truncate table __internal_schema.audit_log""" + + sql """ + create catalog ${credentialCatalog} properties ( + 'type' = 'iceberg', + 'iceberg.catalog.type' = 'rest', + 'uri' = 'http://127.0.0.1:1', + 's3.access_key' = '${accessKey}', + 's3.secret_key' = '${secretKey}', + 's3.endpoint' = 'http://127.0.0.1:1', + 's3.region' = 'us-east-1', + 'iceberg.rest.security.type' = 'oauth2', + 'iceberg.rest.oauth2.credential' = '${oauthCredential}', + 'iceberg.rest.oauth2.server-uri' = 'http://127.0.0.1:1/oauth/tokens' + ) + """ + sql """ + create catalog ${tokenCatalog} properties ( + 'type' = 'iceberg', + 'iceberg.catalog.type' = 'rest', + 'uri' = 'http://127.0.0.1:1', + 'iceberg.rest.security.type' = 'oauth2', + 'iceberg.rest.oauth2.token' = '${oauthToken}' + ) + """ + sql """call flush_audit_log()""" + + // Audit statements are independently persisted after parsing, so they must use the + // same sensitive-key masking contract as SHOW CREATE and printable catalog metadata. + String credentialStmt = waitForAuditStatement(credentialCatalog) + assertFalse(credentialStmt.contains(accessKey)) + assertFalse(credentialStmt.contains(secretKey)) + assertFalse(credentialStmt.contains(oauthCredential)) + assertTrue(credentialStmt.contains('"s3.access_key" = "*XXX"')) + assertTrue(credentialStmt.contains('"s3.secret_key" = "*XXX"')) + assertTrue(credentialStmt.contains('"iceberg.rest.oauth2.credential" = "*XXX"')) + + String tokenStmt = waitForAuditStatement(tokenCatalog) + assertFalse(tokenStmt.contains(oauthToken)) + assertTrue(tokenStmt.contains('"iceberg.rest.oauth2.token" = "*XXX"')) + } finally { + sql """drop catalog if exists ${credentialCatalog}""" + sql """drop catalog if exists ${tokenCatalog}""" + } + }) +} diff --git a/regression-test/suites/compaction/test_table_level_compaction_policy.groovy b/regression-test/suites/compaction/test_table_level_compaction_policy.groovy index 5edff4e38e0da5..0e1c70b02c9eea 100644 --- a/regression-test/suites/compaction/test_table_level_compaction_policy.groovy +++ b/regression-test/suites/compaction/test_table_level_compaction_policy.groovy @@ -219,22 +219,22 @@ suite("test_table_level_compaction_policy") { exception "only time series compaction policy support for time series config" } - test { - sql """ - CREATE TABLE ${tableName} ( - `c_custkey` int(11) NOT NULL COMMENT "", - `c_name` varchar(26) NOT NULL COMMENT "", - `c_address` varchar(41) NOT NULL COMMENT "", - `c_city` varchar(11) NOT NULL COMMENT "" - ) - DUPLICATE KEY (`c_custkey`) - DISTRIBUTED BY HASH(`c_custkey`) BUCKETS 1 - PROPERTIES ( - "replication_num" = "1" - ); - """ - sql """sync""" + sql """ + CREATE TABLE ${tableName} ( + `c_custkey` int(11) NOT NULL COMMENT "", + `c_name` varchar(26) NOT NULL COMMENT "", + `c_address` varchar(41) NOT NULL COMMENT "", + `c_city` varchar(11) NOT NULL COMMENT "" + ) + DUPLICATE KEY (`c_custkey`) + DISTRIBUTED BY HASH(`c_custkey`) BUCKETS 1 + PROPERTIES ( + "replication_num" = "1" + ); + """ + sql """sync""" + test { sql """ alter table ${tableName} set ("compaction_policy" = "ok") """ diff --git a/regression-test/suites/doc/sql-manual/sql-function/test_array_function.groovy b/regression-test/suites/doc/sql-manual/sql-function/test_array_function.groovy index 6f9f1f7b0e0ca4..f20bfb60f38aa9 100644 --- a/regression-test/suites/doc/sql-manual/sql-function/test_array_function.groovy +++ b/regression-test/suites/doc/sql-manual/sql-function/test_array_function.groovy @@ -520,4 +520,4 @@ suite("test_array_function_doc", "p0") { qt_sql """ SELECT ARRAY_SORTBY(x -> x[1], [[1,2],[3,4]]); """ qt_sql """ SELECT ARRAY_SORT([[1,2],[3,4]]); """ qt_sql """ SELECT ARRAY_REVERSE_SORT([[1,2],[3,4]]); """ -} \ No newline at end of file +} diff --git a/regression-test/suites/external_table_p0/iceberg/iceberg_schema_change_ddl.groovy b/regression-test/suites/external_table_p0/iceberg/iceberg_schema_change_ddl.groovy index 1762a088953c72..c197e3f372c656 100644 --- a/regression-test/suites/external_table_p0/iceberg/iceberg_schema_change_ddl.groovy +++ b/regression-test/suites/external_table_p0/iceberg/iceberg_schema_change_ddl.groovy @@ -193,7 +193,7 @@ suite("iceberg_schema_change_ddl", "p0,external,doris,external_docker,external_d sql """ ALTER TABLE ${table_name} MODIFY COLUMN non_col bigint""" exception "Column non_col does not exist" } - // not comment, the comment will be removed + // Omitted COMMENT preserves the existing comment. qt_before_no_comment "desc ${table_name}" sql """ ALTER TABLE ${table_name} MODIFY COLUMN col1 DOUBLE""" qt_after_no_comment "desc ${table_name}" @@ -305,7 +305,7 @@ suite("iceberg_schema_change_ddl", "p0,external,doris,external_docker,external_d // struct/complex type changes test { sql """ ALTER TABLE ${table_name} MODIFY COLUMN address STRING """ - exception "Cannot change column type" + exception "Modify column type from complex to primitive is not supported" } test { diff --git a/regression-test/suites/external_table_p0/iceberg/test_iceberg_jdbc_catalog.groovy b/regression-test/suites/external_table_p0/iceberg/test_iceberg_jdbc_catalog.groovy index 963fa96a13ed27..3ad24e515c683e 100644 --- a/regression-test/suites/external_table_p0/iceberg/test_iceberg_jdbc_catalog.groovy +++ b/regression-test/suites/external_table_p0/iceberg/test_iceberg_jdbc_catalog.groovy @@ -88,6 +88,14 @@ suite("test_iceberg_jdbc_catalog", "p0,external,iceberg,external_docker,external host_ips.add(f[1]) } host_ips = host_ips.unique() + Set localHostIps = ["127.0.0.1", "localhost", "::1"] as Set + java.util.Collections.list(java.net.NetworkInterface.getNetworkInterfaces()).each { networkInterface -> + java.util.Collections.list(networkInterface.getInetAddresses()).each { address -> + localHostIps.add(address.getHostAddress().split("%")[0]) + } + } + localHostIps.add(java.net.InetAddress.getLocalHost().getHostName()) + localHostIps.add(java.net.InetAddress.getLocalHost().getCanonicalHostName()) executeCommand("mkdir -p ${local_driver_dir}", false) if (!new File(local_driver_path).exists()) { @@ -97,9 +105,18 @@ suite("test_iceberg_jdbc_catalog", "p0,external,iceberg,external_docker,external executeCommand("/usr/bin/curl --max-time 600 ${mysql_driver_download_url} --output ${local_mysql_driver_path}", true) } for (def ip in host_ips) { - executeCommand("ssh -o StrictHostKeyChecking=no root@${ip} \"mkdir -p ${jdbc_drivers_dir}\"", false) - scpFiles("root", ip, local_driver_path, jdbc_drivers_dir, false) - scpFiles("root", ip, local_mysql_driver_path, jdbc_drivers_dir, false) + // Scenario: every FE/BE receives the JDBC drivers so distributed scans and FE failover + // do not depend on the regression runner sharing a filesystem with one cluster node. + if (localHostIps.contains(ip)) { + // A local test node must not require root SSH merely to install a JDBC driver. + executeCommand("mkdir -p ${jdbc_drivers_dir}", true) + executeCommand("cp -f ${local_driver_path} ${jdbc_drivers_dir}/${driver_name}", true) + executeCommand("cp -f ${local_mysql_driver_path} ${jdbc_drivers_dir}/${mysql_driver_name}", true) + } else { + executeCommand("ssh -o BatchMode=yes -o StrictHostKeyChecking=no root@${ip} \"mkdir -p ${jdbc_drivers_dir}\"", true) + scpFiles("root", ip, local_driver_path, jdbc_drivers_dir, false) + scpFiles("root", ip, local_mysql_driver_path, jdbc_drivers_dir, false) + } } try { @@ -215,6 +232,33 @@ suite("test_iceberg_jdbc_catalog", "p0,external,iceberg,external_docker,external assertTrue(desc.toString().contains("c_int")) assertTrue(desc.toString().contains("c_string")) + // Scenario TC09-JDBC: schema evolution and historical binding work through JDBC catalog. + String jdbcOldSnapshot = sql(""" + select snapshot_id + from test_datatypes\$snapshots + order by committed_at desc + limit 1 + """)[0][0].toString() + sql """alter table test_datatypes create tag jdbc_before_rename""" + sql """alter table test_datatypes rename column c_string jdbc_renamed_string""" + assertEquals([["hello"], ["world"], ["test"]], sql(""" + select c_string + from test_datatypes for version as of ${jdbcOldSnapshot} + order by c_int + """)) + assertEquals([["hello"], ["world"], ["test"]], sql(""" + select c_string + from test_datatypes@tag(jdbc_before_rename) + order by c_int + """)) + assertEquals([["hello"], ["world"], ["test"]], sql(""" + select jdbc_renamed_string from test_datatypes order by c_int + """)) + test { + sql """select c_string from test_datatypes""" + exception "Unknown column 'c_string'" + } + // Test: INSERT OVERWRITE sql """ INSERT OVERWRITE TABLE test_partitioned diff --git a/regression-test/suites/external_table_p0/iceberg/test_iceberg_multi_spec_filter_aggregate.groovy b/regression-test/suites/external_table_p0/iceberg/test_iceberg_multi_spec_filter_aggregate.groovy new file mode 100644 index 00000000000000..dd09b7be944c33 --- /dev/null +++ b/regression-test/suites/external_table_p0/iceberg/test_iceberg_multi_spec_filter_aggregate.groovy @@ -0,0 +1,120 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +suite("test_iceberg_multi_spec_filter_aggregate", "p0,external") { + String enabled = context.config.otherConfigs.get("enableIcebergTest") + if (enabled == null || !enabled.equalsIgnoreCase("true")) { + logger.info("disable iceberg test.") + return + } + + String catalogName = "test_iceberg_multi_spec_filter_aggregate" + String dbName = "multi_spec_filter_aggregate_db" + String tableName = "multi_spec_filter_aggregate" + String restPort = context.config.otherConfigs.get("iceberg_rest_uri_port") + String minioPort = context.config.otherConfigs.get("iceberg_minio_port") + String externalEnvIp = context.config.otherConfigs.get("externalEnvIp") + + sql """drop catalog if exists ${catalogName}""" + sql """ + create catalog ${catalogName} properties ( + 'type' = 'iceberg', + 'iceberg.catalog.type' = 'rest', + 'uri' = 'http://${externalEnvIp}:${restPort}', + 's3.access_key' = 'admin', + 's3.secret_key' = 'password', + 's3.endpoint' = 'http://${externalEnvIp}:${minioPort}', + 's3.region' = 'us-east-1' + ) + """ + + try { + sql """switch ${catalogName}""" + sql """drop database if exists ${dbName} force""" + sql """create database ${dbName}""" + sql """use ${dbName}""" + sql """set enable_fallback_to_original_planner = false""" + sql """set runtime_filter_mode = 'GLOBAL'""" + + sql """ + create table ${tableName} ( + id int, + code string, + metric int + ) + """ + sql """insert into ${tableName} values (1, 'aa-01', 10), (11, 'bb-11', 110)""" + + sql """alter table ${tableName} add partition key bucket(4, id)""" + sql """insert into ${tableName} values (2, 'aa-02', 20), (12, 'bb-12', 120)""" + + sql """alter table ${tableName} add partition key truncate(2, code)""" + sql """insert into ${tableName} values (3, 'aa-03', 30), (13, 'bb-13', 130)""" + + sql """alter table ${tableName} drop partition key bucket(4, id)""" + sql """insert into ${tableName} values (4, 'aa-04', 40), (14, 'bb-14', 140)""" + + // Every data file must be evaluated with the partition spec that wrote it; applying the + // newest transform to old-spec files can silently prune valid rows or double-count keys. + order_qt_multi_spec_rows """ + select id, code, metric from ${tableName} order by id + """ + order_qt_multi_spec_static_filter """ + select id, code + from ${tableName} + where id in (1, 3, 12, 14) and code >= 'aa-00' + order by id + """ + order_qt_multi_spec_runtime_filter """ + with filter_keys as ( + select 2 as id + union all + select 3 + union all + select 12 + union all + select 14 + ) + select t.id, t.code, t.metric + from ${tableName} t + join filter_keys k on t.id = k.id + order by t.id + """ + order_qt_multi_spec_grouped """ + select substr(code, 1, 2) as code_prefix, + count(*) as row_count, + count(distinct id) as distinct_ids, + sum(metric) as metric_sum + from ${tableName} + group by code_prefix + order by code_prefix + """ + qt_multi_spec_distinct """ + select count(*), count(distinct id), count(distinct code) + from ${tableName} + """ + order_qt_multi_spec_metadata """ + select spec_id, sum(record_count) + from ${tableName}\$partitions + group by spec_id + order by spec_id + """ + } finally { + sql """drop database if exists ${catalogName}.${dbName} force""" + sql """drop catalog if exists ${catalogName}""" + } +} diff --git a/regression-test/suites/external_table_p0/iceberg/test_iceberg_nested_schema_evolution_ddl.groovy b/regression-test/suites/external_table_p0/iceberg/test_iceberg_nested_schema_evolution_ddl.groovy new file mode 100644 index 00000000000000..012bc88e3d9faf --- /dev/null +++ b/regression-test/suites/external_table_p0/iceberg/test_iceberg_nested_schema_evolution_ddl.groovy @@ -0,0 +1,175 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +suite("test_iceberg_nested_schema_evolution_ddl", "p0,external,doris,external_docker,external_docker_doris") { + + String enabled = context.config.otherConfigs.get("enableIcebergTest") + if (enabled == null || !enabled.equalsIgnoreCase("true")) { + logger.info("disable iceberg test.") + return + } + + String restPort = context.config.otherConfigs.get("iceberg_rest_uri_port") + String minioPort = context.config.otherConfigs.get("iceberg_minio_port") + String externalEnvIp = context.config.otherConfigs.get("externalEnvIp") + String catalogName = "test_iceberg_nested_schema_evolution_ddl" + String dbName = "iceberg_nested_schema_evolution_db" + String tableName = "iceberg_nested_evolution" + + sql """drop catalog if exists ${catalogName}""" + sql """ + CREATE CATALOG ${catalogName} PROPERTIES ( + 'type'='iceberg', + 'iceberg.catalog.type'='rest', + 'uri' = 'http://${externalEnvIp}:${restPort}', + "s3.access_key" = "admin", + "s3.secret_key" = "password", + "s3.endpoint" = "http://${externalEnvIp}:${minioPort}", + "s3.region" = "us-east-1" + );""" + + sql """switch ${catalogName};""" + sql """drop database if exists ${dbName} force""" + sql """create database ${dbName}""" + sql """use ${dbName};""" + + sql """set enable_fallback_to_original_planner=false;""" + + sql """drop table if exists ${tableName}""" + sql """ + CREATE TABLE ${tableName} ( + id INT NOT NULL, + s STRUCT, + arr ARRAY>, + m MAP>, + arr_scalar ARRAY, + m_scalar MAP + ); + """ + + sql """ + INSERT INTO ${tableName} VALUES ( + 1, + STRUCT(10, 'old'), + ARRAY(STRUCT(100)), + MAP('k', STRUCT(1000)), + ARRAY(7), + MAP('k', 70) + ) + """ + + sql """ALTER TABLE ${tableName} ADD COLUMN s.c STRING NULL COMMENT 'new nested field'""" + sql """ALTER TABLE ${tableName} ADD COLUMN s.first_pos STRING NULL FIRST""" + sql """ALTER TABLE ${tableName} ADD COLUMN s.after_a STRING NULL AFTER a""" + sql """ALTER TABLE ${tableName} ADD COLUMN arr.element.y INT NULL""" + sql """ALTER TABLE ${tableName} ADD COLUMN arr.element.after_x INT NULL AFTER x""" + sql """ALTER TABLE ${tableName} ADD COLUMN m.value.y INT NULL""" + sql """ALTER TABLE ${tableName} ADD COLUMN m.value.after_v INT NULL AFTER v""" + sql """ALTER TABLE ${tableName} ADD COLUMN s.drop_me STRING NULL""" + sql """ALTER TABLE ${tableName} ADD COLUMN arr.element.drop_me INT NULL""" + sql """ALTER TABLE ${tableName} ADD COLUMN m.value.drop_me INT NULL""" + + test { + sql """ALTER TABLE ${tableName} ADD COLUMN s.required_field INT NOT NULL""" + exception "New nested field 's.required_field' must be nullable" + } + + sql """ALTER TABLE ${tableName} MODIFY COLUMN s.a BIGINT""" + sql """ALTER TABLE ${tableName} MODIFY COLUMN arr.element.x BIGINT""" + sql """ALTER TABLE ${tableName} MODIFY COLUMN m.value.v BIGINT""" + sql """ALTER TABLE ${tableName} MODIFY COLUMN arr_scalar.element BIGINT""" + sql """ALTER TABLE ${tableName} MODIFY COLUMN m_scalar.value BIGINT""" + test { + sql """ALTER TABLE ${tableName} MODIFY COLUMN m_scalar.`key` BIGINT""" + exception "Cannot modify MAP key nested column" + } + + test { + sql """ALTER TABLE ${tableName} MODIFY COLUMN arr_scalar.element COMMENT 'array element comment'""" + exception "Iceberg does not support comments on collection element or value fields" + } + test { + sql """ALTER TABLE ${tableName} MODIFY COLUMN m_scalar.value COMMENT 'map value comment'""" + exception "Iceberg does not support comments on collection element or value fields" + } + test { + sql """ALTER TABLE ${tableName} MODIFY COLUMN arr_scalar.element BIGINT COMMENT 'array element comment'""" + exception "Iceberg does not support comments on collection element or value fields" + } + test { + sql """ALTER TABLE ${tableName} MODIFY COLUMN m_scalar.value BIGINT COMMENT 'map value comment'""" + exception "Iceberg does not support comments on collection element or value fields" + } + test { + sql """ALTER TABLE ${tableName} MODIFY COLUMN arr_scalar.element BIGINT COMMENT ''""" + exception "Iceberg does not support comments on collection element or value fields" + } + test { + sql """ALTER TABLE ${tableName} MODIFY COLUMN m_scalar.value BIGINT COMMENT ''""" + exception "Iceberg does not support comments on collection element or value fields" + } + test { + sql """ALTER TABLE ${tableName} MODIFY COLUMN m_scalar.`key` COMMENT 'map key comment'""" + exception "Cannot modify comment MAP key nested column" + } + + sql """ALTER TABLE ${tableName} RENAME COLUMN s.c TO c2""" + sql """ALTER TABLE ${tableName} RENAME COLUMN arr.element.y TO y2""" + sql """ALTER TABLE ${tableName} RENAME COLUMN m.value.y TO y2""" + sql """ALTER TABLE ${tableName} MODIFY COLUMN s.`c2` COMMENT 'renamed struct field'""" + sql """ALTER TABLE ${tableName} MODIFY COLUMN arr.element.y2 COMMENT 'renamed array element field'""" + sql """ALTER TABLE ${tableName} MODIFY COLUMN m.value.y2 COMMENT 'renamed map value field'""" + sql """ALTER TABLE ${tableName} DROP COLUMN s.drop_me""" + sql """ALTER TABLE ${tableName} DROP COLUMN arr.element.drop_me""" + sql """ALTER TABLE ${tableName} DROP COLUMN m.value.drop_me""" + + sql """ + INSERT INTO ${tableName} VALUES ( + 2, + STRUCT('first', 20, 'after_a', 'new', 'c2'), + ARRAY(STRUCT(200, 202, 201)), + MAP('k', STRUCT(2000, 2002, 2001)), + ARRAY(8), + MAP('k', 80) + ) + """ + + qt_desc """ + SELECT COLUMN_NAME, COLUMN_TYPE + FROM ${catalogName}.information_schema.columns + WHERE TABLE_SCHEMA = '${dbName}' AND TABLE_NAME = '${tableName}' + ORDER BY ORDINAL_POSITION + """ + + order_qt_query_rows """ + SELECT id, + element_at(s, 'first_pos'), + element_at(s, 'a'), + element_at(s, 'after_a'), + element_at(s, 'c2'), + element_at(arr[1], 'x'), + element_at(arr[1], 'after_x'), + element_at(arr[1], 'y2'), + element_at(m['k'], 'v'), + element_at(m['k'], 'after_v'), + element_at(m['k'], 'y2'), + arr_scalar[1], + m_scalar['k'] + FROM ${tableName} + ORDER BY id + """ +} diff --git a/regression-test/suites/external_table_p0/iceberg/test_iceberg_nested_schema_evolution_spark_doris_interop.groovy b/regression-test/suites/external_table_p0/iceberg/test_iceberg_nested_schema_evolution_spark_doris_interop.groovy new file mode 100644 index 00000000000000..96725c2aedd884 --- /dev/null +++ b/regression-test/suites/external_table_p0/iceberg/test_iceberg_nested_schema_evolution_spark_doris_interop.groovy @@ -0,0 +1,553 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +import groovy.json.JsonSlurper + +suite("test_iceberg_nested_schema_evolution_spark_doris_interop", + "p0,external,iceberg,external_docker,external_docker_iceberg") { + String enabled = context.config.otherConfigs.get("enableIcebergTest") + if (enabled == null || !enabled.equalsIgnoreCase("true")) { + logger.info("disable iceberg test.") + return + } + + String catalogName = "test_iceberg_nested_schema_evolution_spark_doris_interop" + String dbName = "iceberg_nested_schema_evolution_interop_db" + String dorisTable = "doris_nested_evolution_to_spark" + String commentTable = "doris_nested_comment_semantics" + String sparkTable = "spark_nested_evolution_to_doris" + String mixedCaseTable = "spark_mixed_case_nested_collision" + String requiredTable = "spark_required_nested_evolution" + String advancedTable = "spark_deep_nested_evolution" + String restPort = context.config.otherConfigs.get("iceberg_rest_uri_port") + String minioPort = context.config.otherConfigs.get("iceberg_minio_port") + String externalEnvIp = context.config.otherConfigs.get("externalEnvIp") + + sql """drop catalog if exists ${catalogName}""" + sql """ + CREATE CATALOG ${catalogName} PROPERTIES ( + 'type'='iceberg', + 'iceberg.catalog.type'='rest', + 'uri' = 'http://${externalEnvIp}:${restPort}', + "s3.access_key" = "admin", + "s3.secret_key" = "password", + "s3.endpoint" = "http://${externalEnvIp}:${minioPort}", + "s3.region" = "us-east-1", + "meta.cache.iceberg.table.ttl-second" = "0" + ); + """ + + sql """switch ${catalogName}""" + sql """drop database if exists ${dbName} force""" + sql """create database ${dbName}""" + sql """use ${dbName}""" + sql """set enable_fallback_to_original_planner=false""" + + spark_iceberg """CREATE DATABASE IF NOT EXISTS demo.${dbName}""" + + sql """ + CREATE TABLE ${commentTable} ( + id INT, + info STRUCT< + metric:INT COMMENT 'metric doc', + note:STRING COMMENT 'old note', + clear_me:STRING COMMENT 'clear doc', + payload:STRUCT< + name:STRING COMMENT 'name doc', + count:INT COMMENT 'count doc' + > COMMENT 'payload doc' + > + ) + """ + sql """ALTER TABLE ${commentTable} MODIFY COLUMN info.note COMMENT 'new note'""" + sql """ALTER TABLE ${commentTable} MODIFY COLUMN info.metric BIGINT""" + sql """ALTER TABLE ${commentTable} MODIFY COLUMN info.clear_me STRING COMMENT ''""" + sql """ + ALTER TABLE ${commentTable} MODIFY COLUMN info.payload + STRUCT + """ + + String loadTableUrl = "http://${externalEnvIp}:${restPort}/v1/namespaces/${dbName}/tables/${commentTable}" + def loadTableResponse = new JsonSlurper().parseText(new URL(loadTableUrl).getText("UTF-8")) + def tableMetadata = loadTableResponse.metadata + def currentSchema = tableMetadata.schemas.find { + it["schema-id"] == tableMetadata["current-schema-id"] + } + assertNotNull(currentSchema, "current schema should exist in Iceberg metadata") + def infoColumn = currentSchema.fields.find { it.name == "info" } + assertNotNull(infoColumn, "info column should exist in Iceberg metadata") + def infoFields = infoColumn.type.fields.collectEntries { [(it.name): it] } + assertEquals("long", infoFields.metric.type) + assertEquals("metric doc", infoFields.metric.doc) + assertEquals("new note", infoFields.note.doc) + assertTrue(infoFields.clear_me.doc == null || infoFields.clear_me.doc == "") + assertEquals("payload doc", infoFields.payload.doc) + def payloadFields = infoFields.payload.type.fields.collectEntries { [(it.name): it] } + assertTrue(payloadFields.name.doc == null || payloadFields.name.doc == "") + assertEquals("long", payloadFields.count.type) + assertEquals("count doc", payloadFields.count.doc) + + sql """ + CREATE TABLE ${dorisTable} ( + id INT NOT NULL, + info STRUCT, + events ARRAY>, + attrs MAP> + ) + """ + sql """ + INSERT INTO ${dorisTable} VALUES ( + 1, + STRUCT(10, 'doris_before'), + ARRAY(STRUCT(100)), + MAP('k', STRUCT(1000)) + ) + """ + + sql """ALTER TABLE ${dorisTable} ADD COLUMN info.doris_added STRING NULL COMMENT 'added by doris'""" + sql """ALTER TABLE ${dorisTable} ADD COLUMN info.doris_first STRING NULL FIRST""" + sql """ALTER TABLE ${dorisTable} ADD COLUMN info.doris_after_metric STRING NULL AFTER metric""" + sql """ALTER TABLE ${dorisTable} ADD COLUMN events.element.doris_score INT NULL""" + sql """ALTER TABLE ${dorisTable} ADD COLUMN events.element.doris_first_score INT NULL FIRST""" + sql """ALTER TABLE ${dorisTable} ADD COLUMN events.element.doris_after_score INT NULL AFTER score""" + sql """ALTER TABLE ${dorisTable} ADD COLUMN attrs.value.doris_code INT NULL""" + sql """ALTER TABLE ${dorisTable} ADD COLUMN attrs.value.doris_first_code INT NULL FIRST""" + sql """ALTER TABLE ${dorisTable} ADD COLUMN attrs.value.doris_after_code INT NULL AFTER code""" + sql """ALTER TABLE ${dorisTable} ADD COLUMN info.drop_me STRING NULL""" + sql """ALTER TABLE ${dorisTable} ADD COLUMN events.element.drop_me INT NULL""" + sql """ALTER TABLE ${dorisTable} ADD COLUMN attrs.value.drop_me INT NULL""" + sql """ALTER TABLE ${dorisTable} MODIFY COLUMN info.metric BIGINT""" + sql """ALTER TABLE ${dorisTable} MODIFY COLUMN events.element.score BIGINT""" + sql """ALTER TABLE ${dorisTable} MODIFY COLUMN attrs.value.code BIGINT""" + sql """ALTER TABLE ${dorisTable} RENAME COLUMN info.doris_added TO doris_renamed""" + sql """ALTER TABLE ${dorisTable} RENAME COLUMN events.element.doris_score TO doris_score_renamed""" + sql """ALTER TABLE ${dorisTable} RENAME COLUMN attrs.value.doris_code TO doris_code_renamed""" + sql """ALTER TABLE ${dorisTable} MODIFY COLUMN info.doris_renamed COMMENT 'renamed by doris'""" + sql """ALTER TABLE ${dorisTable} MODIFY COLUMN events.element.doris_score_renamed COMMENT 'renamed by doris'""" + sql """ALTER TABLE ${dorisTable} MODIFY COLUMN attrs.value.doris_code_renamed COMMENT 'renamed by doris'""" + sql """ALTER TABLE ${dorisTable} DROP COLUMN info.drop_me""" + sql """ALTER TABLE ${dorisTable} DROP COLUMN events.element.drop_me""" + sql """ALTER TABLE ${dorisTable} DROP COLUMN attrs.value.drop_me""" + + spark_iceberg """REFRESH TABLE demo.${dbName}.${dorisTable}""" + spark_iceberg """ + INSERT INTO demo.${dbName}.${dorisTable} VALUES ( + 2, + NAMED_STRUCT('doris_first', 'spark_first', + 'metric', CAST(20 AS BIGINT), + 'doris_after_metric', 'spark_after_metric', + 'label', 'spark_after_doris', + 'doris_renamed', 'spark_can_write_doris_field'), + ARRAY(NAMED_STRUCT('doris_first_score', 202, + 'score', CAST(200 AS BIGINT), + 'doris_after_score', 203, + 'doris_score_renamed', 201)), + MAP('k', NAMED_STRUCT('doris_first_code', 2002, + 'code', CAST(2000 AS BIGINT), + 'doris_after_code', 2003, + 'doris_code_renamed', 2001)) + ) + """ + + sql """refresh table ${dbName}.${dorisTable}""" + def dorisDrivenSparkRows = spark_iceberg """ + SELECT id, + info.doris_first, + info.metric, + info.doris_after_metric, + info.label, + info.doris_renamed, + events[0].doris_first_score, + events[0].score, + events[0].doris_after_score, + events[0].doris_score_renamed, + attrs['k'].doris_first_code, + attrs['k'].code, + attrs['k'].doris_after_code, + attrs['k'].doris_code_renamed + FROM demo.${dbName}.${dorisTable} + ORDER BY id + """ + String dorisDrivenDorisQuery = """ + SELECT id, + element_at(info, 'doris_first'), + element_at(info, 'metric'), + element_at(info, 'doris_after_metric'), + element_at(info, 'label'), + element_at(info, 'doris_renamed'), + element_at(events[1], 'doris_first_score'), + element_at(events[1], 'score'), + element_at(events[1], 'doris_after_score'), + element_at(events[1], 'doris_score_renamed'), + element_at(attrs['k'], 'doris_first_code'), + element_at(attrs['k'], 'code'), + element_at(attrs['k'], 'doris_after_code'), + element_at(attrs['k'], 'doris_code_renamed') + FROM ${dorisTable} + ORDER BY id + """ + order_qt_doris_driven_rows dorisDrivenDorisQuery + def dorisDrivenDorisRows = sql dorisDrivenDorisQuery + assertSparkDorisResultEquals(dorisDrivenSparkRows, dorisDrivenDorisRows) + + spark_iceberg_multi """ + DROP TABLE IF EXISTS demo.${dbName}.${sparkTable}; + DROP TABLE IF EXISTS demo.${dbName}.${mixedCaseTable}; + CREATE TABLE demo.${dbName}.${mixedCaseTable} ( + Id INT, + Label STRING, + Info STRUCT + ) USING iceberg; + CREATE TABLE demo.${dbName}.${sparkTable} ( + id INT, + info STRUCT, + events ARRAY>, + attrs MAP> + ) USING iceberg; + INSERT INTO demo.${dbName}.${sparkTable} VALUES ( + 1, + NAMED_STRUCT('metric', 10, 'label', 'spark_before'), + ARRAY(NAMED_STRUCT('score', 100)), + MAP('k', NAMED_STRUCT('code', 1000)) + ); + ALTER TABLE demo.${dbName}.${sparkTable} ADD COLUMN info.spark_added STRING; + ALTER TABLE demo.${dbName}.${sparkTable} ADD COLUMN info.spark_first STRING FIRST; + ALTER TABLE demo.${dbName}.${sparkTable} ADD COLUMN info.spark_after_metric STRING AFTER metric; + ALTER TABLE demo.${dbName}.${sparkTable} ADD COLUMN events.element.spark_score INT; + ALTER TABLE demo.${dbName}.${sparkTable} ADD COLUMN events.element.spark_first_score INT FIRST; + ALTER TABLE demo.${dbName}.${sparkTable} ADD COLUMN events.element.spark_after_score INT AFTER score; + ALTER TABLE demo.${dbName}.${sparkTable} ADD COLUMN attrs.value.spark_code INT; + ALTER TABLE demo.${dbName}.${sparkTable} ADD COLUMN attrs.value.spark_first_code INT FIRST; + ALTER TABLE demo.${dbName}.${sparkTable} ADD COLUMN attrs.value.spark_after_code INT AFTER code; + ALTER TABLE demo.${dbName}.${sparkTable} ADD COLUMN info.drop_me STRING; + ALTER TABLE demo.${dbName}.${sparkTable} ADD COLUMN events.element.drop_me INT; + ALTER TABLE demo.${dbName}.${sparkTable} ADD COLUMN attrs.value.drop_me INT; + ALTER TABLE demo.${dbName}.${sparkTable} ALTER COLUMN info.metric TYPE BIGINT; + ALTER TABLE demo.${dbName}.${sparkTable} ALTER COLUMN events.element.score TYPE BIGINT; + ALTER TABLE demo.${dbName}.${sparkTable} ALTER COLUMN attrs.value.code TYPE BIGINT; + ALTER TABLE demo.${dbName}.${sparkTable} RENAME COLUMN info.spark_added TO spark_renamed; + ALTER TABLE demo.${dbName}.${sparkTable} RENAME COLUMN events.element.spark_score TO spark_score_renamed; + ALTER TABLE demo.${dbName}.${sparkTable} RENAME COLUMN attrs.value.spark_code TO spark_code_renamed; + ALTER TABLE demo.${dbName}.${sparkTable} ALTER COLUMN info.spark_renamed COMMENT 'renamed by spark'; + ALTER TABLE demo.${dbName}.${sparkTable} ALTER COLUMN events.element.spark_score_renamed COMMENT 'renamed by spark'; + ALTER TABLE demo.${dbName}.${sparkTable} ALTER COLUMN attrs.value.spark_code_renamed COMMENT 'renamed by spark'; + ALTER TABLE demo.${dbName}.${sparkTable} DROP COLUMN info.drop_me; + ALTER TABLE demo.${dbName}.${sparkTable} DROP COLUMN events.element.drop_me; + ALTER TABLE demo.${dbName}.${sparkTable} DROP COLUMN attrs.value.drop_me; + INSERT INTO demo.${dbName}.${sparkTable} VALUES ( + 2, + NAMED_STRUCT('spark_first', 'spark_first_field', + 'metric', CAST(20 AS BIGINT), + 'spark_after_metric', 'spark_after_metric_field', + 'label', 'spark_after', + 'spark_renamed', 'spark_new_field'), + ARRAY(NAMED_STRUCT('spark_first_score', 202, + 'score', CAST(200 AS BIGINT), + 'spark_after_score', 203, + 'spark_score_renamed', 201)), + MAP('k', NAMED_STRUCT('spark_first_code', 2002, + 'code', CAST(2000 AS BIGINT), + 'spark_after_code', 2003, + 'spark_code_renamed', 2001)) + ); + """ + + sql """refresh catalog ${catalogName}""" + sql """switch ${catalogName}""" + sql """use ${dbName}""" + + test { + sql """ALTER TABLE ${mixedCaseTable} ADD COLUMN id STRING NULL""" + exception "conflicts with existing Iceberg field 'Id' (case-insensitive)" + } + test { + sql """ALTER TABLE ${mixedCaseTable} ADD COLUMN (id STRING)""" + exception "conflicts with existing Iceberg field 'Id' (case-insensitive)" + } + test { + sql """ALTER TABLE ${mixedCaseTable} ADD COLUMN (new_field STRING, NEW_FIELD STRING)""" + exception "conflicts with another requested column (case-insensitive)" + } + test { + sql """ALTER TABLE ${mixedCaseTable} RENAME COLUMN label TO id""" + exception "conflicts with existing Iceberg field 'Id' (case-insensitive)" + } + sql """ALTER TABLE ${mixedCaseTable} RENAME COLUMN label TO label""" + + test { + sql """ALTER TABLE ${mixedCaseTable} ADD COLUMN info.metric STRING NULL""" + exception "conflicts with existing Iceberg field 'Info.Metric' (case-insensitive)" + } + test { + sql """ALTER TABLE ${mixedCaseTable} RENAME COLUMN info.label TO metric""" + exception "conflicts with existing Iceberg field 'Info.Metric' (case-insensitive)" + } + + qt_spark_driven_schema """ + SELECT COLUMN_NAME, COLUMN_TYPE + FROM ${catalogName}.information_schema.columns + WHERE TABLE_SCHEMA = '${dbName}' AND TABLE_NAME = '${sparkTable}' + ORDER BY ORDINAL_POSITION + """ + + String sparkDrivenDorisQuery = """ + SELECT id, + element_at(info, 'spark_first'), + element_at(info, 'metric'), + element_at(info, 'spark_after_metric'), + element_at(info, 'label'), + element_at(info, 'spark_renamed'), + element_at(events[1], 'spark_first_score'), + element_at(events[1], 'score'), + element_at(events[1], 'spark_after_score'), + element_at(events[1], 'spark_score_renamed'), + element_at(attrs['k'], 'spark_first_code'), + element_at(attrs['k'], 'code'), + element_at(attrs['k'], 'spark_after_code'), + element_at(attrs['k'], 'spark_code_renamed') + FROM ${sparkTable} + ORDER BY id + """ + order_qt_spark_driven_rows_before_write sparkDrivenDorisQuery + + sql """ + INSERT INTO ${sparkTable} VALUES ( + 3, + STRUCT('doris_first_field', 30, 'doris_after_metric_field', + 'doris_after_spark', 'doris_can_write_spark_field'), + ARRAY(STRUCT(302, 300, 303, 301)), + MAP('k', STRUCT(3002, 3000, 3003, 3001)) + ) + """ + + spark_iceberg """REFRESH TABLE demo.${dbName}.${sparkTable}""" + def sparkDrivenSparkRows = spark_iceberg """ + SELECT id, + info.spark_first, + info.metric, + info.spark_after_metric, + info.label, + info.spark_renamed, + events[0].spark_first_score, + events[0].score, + events[0].spark_after_score, + events[0].spark_score_renamed, + attrs['k'].spark_first_code, + attrs['k'].code, + attrs['k'].spark_after_code, + attrs['k'].spark_code_renamed + FROM demo.${dbName}.${sparkTable} + ORDER BY id + """ + order_qt_spark_driven_rows_after_write sparkDrivenDorisQuery + def sparkDrivenDorisRows = sql sparkDrivenDorisQuery + assertSparkDorisResultEquals(sparkDrivenSparkRows, sparkDrivenDorisRows) + + spark_iceberg_multi """ + DROP TABLE IF EXISTS demo.${dbName}.${requiredTable}; + CREATE TABLE demo.${dbName}.${requiredTable} ( + id INT, + info STRUCT + ) USING iceberg; + INSERT INTO demo.${dbName}.${requiredTable} VALUES ( + 1, + NAMED_STRUCT('required_metric', 10, 'required_label', 'old-label') + ); + """ + + sql """refresh catalog ${catalogName}""" + sql """switch ${catalogName}""" + sql """use ${dbName}""" + + sql """ALTER TABLE ${requiredTable} MODIFY COLUMN info.required_metric BIGINT""" + sql """ALTER TABLE ${requiredTable} MODIFY COLUMN info.required_label STRING NULL""" + test { + sql """ALTER TABLE ${requiredTable} MODIFY COLUMN info.required_label STRING NOT NULL""" + exception "Can not change nullable column info.required_label to not null" + } + + String requiredTableUrl = + "http://${externalEnvIp}:${restPort}/v1/namespaces/${dbName}/tables/${requiredTable}" + def requiredTableMetadata = new JsonSlurper().parseText( + new URL(requiredTableUrl).getText("UTF-8")).metadata + def requiredSchema = requiredTableMetadata.schemas.find { + it["schema-id"] == requiredTableMetadata["current-schema-id"] + } + def requiredInfo = requiredSchema.fields.find { it.name == "info" } + def requiredInfoFields = requiredInfo.type.fields.collectEntries { [(it.name): it] } + assertEquals("long", requiredInfoFields.required_metric.type) + assertTrue(requiredInfoFields.required_metric.required) + assertFalse(requiredInfoFields.required_label.required) + + sql """ + INSERT INTO ${requiredTable} VALUES ( + 2, + STRUCT(20, NULL) + ) + """ + + String requiredDorisQuery = """ + SELECT id, + element_at(info, 'required_metric'), + element_at(info, 'required_label') + FROM ${requiredTable} + ORDER BY id + """ + order_qt_required_nested_rows requiredDorisQuery + spark_iceberg """REFRESH TABLE demo.${dbName}.${requiredTable}""" + def requiredSparkRows = spark_iceberg """ + SELECT id, info.required_metric, info.required_label + FROM demo.${dbName}.${requiredTable} + ORDER BY id + """ + def requiredDorisRows = sql requiredDorisQuery + assertSparkDorisResultEquals(requiredSparkRows, requiredDorisRows) + + spark_iceberg_multi """ + DROP TABLE IF EXISTS demo.${dbName}.${advancedTable}; + CREATE TABLE demo.${dbName}.${advancedTable} ( + id INT, + root STRUCT< + first_field: STRING, + middle_field: STRING, + last_field: STRING, + items: ARRAY>>> + > + ) USING iceberg; + INSERT INTO demo.${dbName}.${advancedTable} VALUES ( + 1, + NAMED_STRUCT( + 'first_field', 'first-old', + 'middle_field', 'middle-old', + 'last_field', 'last-old', + 'items', ARRAY(NAMED_STRUCT( + 'attrs', MAP('k', NAMED_STRUCT( + 'score', CAST(1.5 AS FLOAT), + 'amount', CAST(12.34 AS DECIMAL(9, 2)), + 'note', 'old-note' + )) + )) + ) + ); + """ + + sql """refresh catalog ${catalogName}""" + sql """switch ${catalogName}""" + sql """use ${dbName}""" + + sql """ALTER TABLE ${advancedTable} MODIFY COLUMN root.last_field STRING FIRST""" + sql """ALTER TABLE ${advancedTable} MODIFY COLUMN root.first_field STRING AFTER middle_field""" + sql """ALTER TABLE ${advancedTable} MODIFY COLUMN root.items.element.attrs.value.score DOUBLE""" + sql """ALTER TABLE ${advancedTable} MODIFY COLUMN root.items.element.attrs.value.amount DECIMAL(18, 2)""" + test { + sql """ALTER TABLE ${advancedTable} + MODIFY COLUMN root.items.element.attrs.value.amount DECIMAL(18, 3)""" + exception "Cannot change column type" + } + test { + sql """ALTER TABLE ${advancedTable} + MODIFY COLUMN root.items.element.attrs.value.score FLOAT""" + exception "Cannot change column type" + } + sql """ALTER TABLE ${advancedTable} + ADD COLUMN root.items.element.attrs.value.added STRING NULL AFTER amount""" + sql """ALTER TABLE ${advancedTable} + RENAME COLUMN root.items.element.attrs.value.added TO renamed""" + sql """ALTER TABLE ${advancedTable} + MODIFY COLUMN root.items.element.attrs.value.renamed COMMENT 'deep field'""" + sql """ALTER TABLE ${advancedTable} + MODIFY COLUMN root.items.element.attrs.value.note STRING FIRST""" + + String advancedTableUrl = + "http://${externalEnvIp}:${restPort}/v1/namespaces/${dbName}/tables/${advancedTable}" + def advancedTableMetadata = new JsonSlurper().parseText( + new URL(advancedTableUrl).getText("UTF-8")).metadata + def advancedSchema = advancedTableMetadata.schemas.find { + it["schema-id"] == advancedTableMetadata["current-schema-id"] + } + def rootField = advancedSchema.fields.find { it.name == "root" } + assertEquals(["last_field", "middle_field", "first_field", "items"], + rootField.type.fields.collect { it.name }) + def itemsField = rootField.type.fields.find { it.name == "items" } + def attrsField = itemsField.type.element.fields.find { it.name == "attrs" } + def deepFields = attrsField.type.value.fields + assertEquals(["note", "score", "amount", "renamed"], deepFields.collect { it.name }) + assertEquals("double", deepFields.find { it.name == "score" }.type) + assertEquals("decimal(18, 2)", deepFields.find { it.name == "amount" }.type) + assertEquals("deep field", deepFields.find { it.name == "renamed" }.doc) + + sql """ALTER TABLE ${advancedTable} + DROP COLUMN root.items.element.attrs.value.renamed""" + + def droppedFieldMetadata = new JsonSlurper().parseText( + new URL(advancedTableUrl).getText("UTF-8")).metadata + def droppedFieldSchema = droppedFieldMetadata.schemas.find { + it["schema-id"] == droppedFieldMetadata["current-schema-id"] + } + def droppedRootField = droppedFieldSchema.fields.find { it.name == "root" } + def droppedItemsField = droppedRootField.type.fields.find { it.name == "items" } + def droppedAttrsField = droppedItemsField.type.element.fields.find { it.name == "attrs" } + assertEquals(["note", "score", "amount"], + droppedAttrsField.type.value.fields.collect { it.name }) + + sql """ + INSERT INTO ${advancedTable} VALUES ( + 2, + STRUCT( + 'last-new', + 'middle-new', + 'first-new', + ARRAY(STRUCT(MAP('k', STRUCT( + 'new-note', + CAST(2.5 AS DOUBLE), + CAST(56.78 AS DECIMAL(18, 2)) + )))) + ) + ) + """ + + String advancedDorisQuery = """ + SELECT id, + element_at(root, 'last_field'), + element_at(root, 'middle_field'), + element_at(root, 'first_field'), + element_at(element_at(element_at(root, 'items')[1], 'attrs')['k'], 'note'), + element_at(element_at(element_at(root, 'items')[1], 'attrs')['k'], 'score'), + element_at(element_at(element_at(root, 'items')[1], 'attrs')['k'], 'amount') + FROM ${advancedTable} + ORDER BY id + """ + order_qt_deep_nested_rows advancedDorisQuery + spark_iceberg """REFRESH TABLE demo.${dbName}.${advancedTable}""" + def advancedSparkRows = spark_iceberg """ + SELECT id, + root.last_field, + root.middle_field, + root.first_field, + root.items[0].attrs['k'].note, + root.items[0].attrs['k'].score, + root.items[0].attrs['k'].amount + FROM demo.${dbName}.${advancedTable} + ORDER BY id + """ + def advancedDorisRows = sql advancedDorisQuery + assertSparkDorisResultEquals(advancedSparkRows, advancedDorisRows) +} diff --git a/regression-test/suites/external_table_p0/iceberg/test_iceberg_partition_evolution_equality_delete.groovy b/regression-test/suites/external_table_p0/iceberg/test_iceberg_partition_evolution_equality_delete.groovy new file mode 100644 index 00000000000000..fab5dc7176df10 --- /dev/null +++ b/regression-test/suites/external_table_p0/iceberg/test_iceberg_partition_evolution_equality_delete.groovy @@ -0,0 +1,302 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +suite("test_iceberg_partition_evolution_equality_delete", + "p0,external,iceberg,external_docker,external_docker_iceberg") { + String enabled = context.config.otherConfigs.get("enableIcebergTest") + if (enabled == null || !enabled.equalsIgnoreCase("true")) { + logger.info("disable iceberg test") + return + } + + String restPort = context.config.otherConfigs.get("iceberg_rest_uri_port") + String minioPort = context.config.otherConfigs.get("iceberg_minio_port") + String externalEnvIp = context.config.otherConfigs.get("externalEnvIp") + String catalogName = "test_iceberg_partition_evolution_equality_delete" + String dbName = "iceberg_partition_evolution_equality_delete_db" + String tableName = "equality_delete_evolved" + + def stringRows = { String query -> + sql(query).collect { row -> + row.collect { value -> value == null ? null : value.toString() } + } + } + def latestSnapshotId = { + // The Java helper commits through the REST catalog directly, so invalidate Spark's + // cached table metadata before recording the snapshot used by time-travel assertions. + spark_iceberg """refresh table demo.${dbName}.${tableName}""" + return spark_iceberg(""" + select snapshot_id + from demo.${dbName}.${tableName}.snapshots + order by committed_at desc + limit 1 + """)[0][0].toString() + } + def executeCommand = { String command, int timeoutSeconds = 300 -> + StringBuilder stdout = new StringBuilder() + StringBuilder stderr = new StringBuilder() + def process = new ProcessBuilder("/bin/bash", "-c", command).start() + process.consumeProcessOutput(stdout, stderr) + process.waitForOrKill(timeoutSeconds * 1000) + assertEquals(0, process.exitValue(), + "Command failed\nstdout:\n${stdout}\nstderr:\n${stderr}") + return stdout.toString() + } + String dockerCommand = context.config.otherConfigs.get("externalDockerCommand") ?: "docker" + String sparkContainer = context.config.otherConfigs.get("icebergSparkContainer") + if (sparkContainer == null || sparkContainer.isEmpty()) { + String containers = executeCommand( + "${dockerCommand} ps --format '{{.ID}}\t{{.Names}}'", 30) + def matches = [] + containers.readLines().each { String line -> + String containerId = line.split(/\t/, 2)[0] + String probe = "${dockerCommand} exec ${containerId} bash -lc " + + "'test -f /mnt/SUCCESS && command -v spark-sql >/dev/null'" + try { + executeCommand(probe, 30) + matches.add(containerId) + } catch (Throwable ignored) { + // A shared external environment contains multiple services; only Spark has the + // Iceberg jars needed to construct a real equality-delete file. + } + } + assertEquals(1, matches.size(), "Expected exactly one usable Spark Iceberg container") + sparkContainer = matches[0] + } + def runInSparkContainer = { String command -> + executeCommand("${dockerCommand} exec ${sparkContainer} bash -lc '${command}'", 300) + } + + String javaSource = ''' +import java.util.HashMap; +import java.util.Map; +import org.apache.iceberg.CatalogUtil; +import org.apache.iceberg.DeleteFile; +import org.apache.iceberg.PartitionSpec; +import org.apache.iceberg.Schema; +import org.apache.iceberg.Table; +import org.apache.iceberg.catalog.Catalog; +import org.apache.iceberg.catalog.TableIdentifier; +import org.apache.iceberg.data.GenericRecord; +import org.apache.iceberg.data.Record; +import org.apache.iceberg.data.parquet.GenericParquetWriter; +import org.apache.iceberg.deletes.EqualityDeleteWriter; +import org.apache.iceberg.io.OutputFile; +import org.apache.iceberg.parquet.Parquet; + +public class AppendEvolutionEqualityDelete { + public static void main(String[] args) throws Exception { + Map props = new HashMap<>(); + props.put("type", "rest"); + props.put("uri", "http://rest:8181"); + props.put("warehouse", "s3://warehouse/wh/"); + props.put("io-impl", "org.apache.iceberg.aws.s3.S3FileIO"); + props.put("s3.endpoint", "http://minio:9000"); + props.put("s3.path-style-access", "true"); + props.put("s3.region", "us-east-1"); + Catalog catalog = CatalogUtil.buildIcebergCatalog("demo", props, null); + Table table = catalog.loadTable(TableIdentifier.of(args[0], args[1])); + if (!table.spec().isUnpartitioned()) { + throw new IllegalStateException("Equality-delete checkpoints must use an unpartitioned spec"); + } + String fieldName = args[2]; + Schema equalitySchema = table.schema().select(fieldName); + int fieldId = table.schema().findField(fieldName).fieldId(); + OutputFile output = table.io().newOutputFile( + table.location() + "/data/evolution-equality-delete-" + args[3] + + "-" + System.currentTimeMillis() + ".parquet"); + EqualityDeleteWriter writer = Parquet.writeDeletes(output) + .forTable(table) + .rowSchema(equalitySchema) + .withSpec(PartitionSpec.unpartitioned()) + .createWriterFunc(GenericParquetWriter::create) + .equalityFieldIds(fieldId) + .overwrite() + .buildEqualityWriter(); + GenericRecord record = GenericRecord.create(equalitySchema); + record.setField(fieldName, Integer.valueOf(args[3])); + writer.write(record); + writer.close(); + DeleteFile deleteFile = writer.toDeleteFile(); + table.newRowDelta().addDeletes(deleteFile).commit(); + } +} +''' + String encodedJava = javaSource.getBytes("UTF-8").encodeBase64().toString() + + sql """drop catalog if exists ${catalogName}""" + sql """ + create catalog ${catalogName} properties ( + 'type'='iceberg', + 'iceberg.catalog.type'='rest', + 'uri'='http://${externalEnvIp}:${restPort}', + 's3.access_key'='admin', + 's3.secret_key'='password', + 's3.endpoint'='http://${externalEnvIp}:${minioPort}', + 's3.region'='us-east-1', + 'meta.cache.iceberg.table.ttl-second'='0' + ) + """ + + try { + runInSparkContainer( + "echo ${encodedJava} | base64 -d >/tmp/AppendEvolutionEqualityDelete.java && " + + "javac -cp \"/opt/spark/jars/*\" " + + "/tmp/AppendEvolutionEqualityDelete.java") + spark_iceberg_multi """ + create database if not exists demo.${dbName}; + drop table if exists demo.${dbName}.${tableName}; + create table demo.${dbName}.${tableName} ( + id int, + category string, + code string, + event_time timestamp, + payload struct + ) using iceberg + tblproperties ( + 'format-version'='2', + 'write.format.default'='parquet' + ); + insert into demo.${dbName}.${tableName} values + (1, 'A', 'aa-1', timestamp '2026-01-01 01:00:00', + named_struct('metric', 10, 'label', 'base-a')), + (2, 'A', 'aa-2', timestamp '2026-01-01 02:00:00', + named_struct('metric', 20, 'label', 'base-delete')), + (3, 'B', 'bb-1', timestamp '2026-01-02 01:00:00', + named_struct('metric', 30, 'label', 'base-b')), + (4, 'C', 'cc-1', timestamp '2026-01-03 01:00:00', + named_struct('metric', 40, 'label', 'base-c')); + """ + String baseSnapshot = latestSnapshotId() + sql """ + alter table `${catalogName}`.`${dbName}`.`${tableName}` + create tag equality_base as of version ${baseSnapshot} + """ + + // Scenario PE-EQ01: an equality delete written under the original unpartitioned spec + // must remain effective after later partition specs are added. + runInSparkContainer( + "java -cp \"/tmp:/opt/spark/jars/*\" AppendEvolutionEqualityDelete " + + "${dbName} ${tableName} id 2") + String firstDeleteSnapshot = latestSnapshotId() + + // Scenario PE-EQ02: ADD identity/day/bucket fields and a nested child after the delete. + spark_iceberg_multi """ + alter table demo.${dbName}.${tableName} add partition field category; + alter table demo.${dbName}.${tableName} add partition field days(event_time); + alter table demo.${dbName}.${tableName} add partition field bucket(8, id); + alter table demo.${dbName}.${tableName} add column payload.extra string; + insert into demo.${dbName}.${tableName} values + (5, 'A', 'aa-3', timestamp '2026-02-01 01:00:00', + named_struct('metric', 50, 'label', 'added-a', 'extra', 'new-child')), + (6, 'A', 'aa-4', timestamp '2026-02-01 02:00:00', + named_struct('metric', 60, 'label', 'added-delete', 'extra', 'new-child')); + """ + String addedSnapshot = latestSnapshotId() + + // Scenario PE-EQ03: REPLACE day -> month while renaming/promoting nested fields. + spark_iceberg_multi """ + alter table demo.${dbName}.${tableName} + replace partition field days(event_time) with months(event_time); + alter table demo.${dbName}.${tableName} + replace partition field bucket(8, id) with truncate(2, code); + alter table demo.${dbName}.${tableName} + rename column payload.label to renamed_label; + alter table demo.${dbName}.${tableName} + alter column payload.metric type bigint; + insert into demo.${dbName}.${tableName} values + (7, 'A', 'aa-5', timestamp '2026-03-01 01:00:00', + named_struct('metric', 7000000000, + 'renamed_label', 'replace-a', 'extra', 'renamed-child')), + (8, 'A', 'aa-6', timestamp '2026-03-01 02:00:00', + named_struct('metric', 80, + 'renamed_label', 'replace-delete', 'extra', 'renamed-child')); + """ + + // Scenario PE-EQ04: return to an unpartitioned spec and write equality deletes after + // evolution. This covers delete files both before and after the multi-spec interval. + spark_iceberg_multi """ + alter table demo.${dbName}.${tableName} drop partition field category; + alter table demo.${dbName}.${tableName} drop partition field months(event_time); + alter table demo.${dbName}.${tableName} drop partition field truncate(2, code); + """ + runInSparkContainer( + "java -cp \"/tmp:/opt/spark/jars/*\" AppendEvolutionEqualityDelete " + + "${dbName} ${tableName} id 6") + runInSparkContainer( + "java -cp \"/tmp:/opt/spark/jars/*\" AppendEvolutionEqualityDelete " + + "${dbName} ${tableName} id 8") + String finalSnapshot = latestSnapshotId() + sql """ + alter table `${catalogName}`.`${dbName}`.`${tableName}` + create tag equality_final as of version ${finalSnapshot} + """ + + sql """switch ${catalogName}""" + sql """use ${dbName}""" + sql """refresh catalog ${catalogName}""" + + // Scenario PE-EQ05: partition-source filters honor equality deletes across every spec. + qt_current_filter """ + select id from ${tableName} where category = 'A' order by id + """ + qt_current_nested """ + select id, payload.metric from ${tableName} + where event_time >= timestamp '2026-03-01 00:00:00' + and payload.metric > 5000000000 + order by id + """ + + // Scenario PE-EQ06: numeric snapshots and tags select the matching delete/spec state. + qt_base_tag """ + select id from ${tableName}@tag(equality_base) + where category = 'A' order by id + """ + qt_first_delete_snapshot """ + select id from ${tableName} for version as of ${firstDeleteSnapshot} + where category = 'A' order by id + """ + qt_added_snapshot """ + select id from ${tableName} for version as of ${addedSnapshot} + where category = 'A' order by id + """ + qt_final_tag """ + select id from ${tableName}@tag(equality_final) + where category = 'A' order by id + """ + + // Scenario PE-EQ07: both scanner implementations apply the same equality deletes. + sql """set enable_file_scanner_v2=true""" + qt_scanner_v2 """ + select id from ${tableName} where category = 'A' order by id + """ + sql """set enable_file_scanner_v2=false""" + qt_scanner_v1 """ + select id from ${tableName} where category = 'A' order by id + """ + + List> equalityFiles = stringRows(""" + select file_format from ${tableName}\$all_files + where content = 2 order by file_path + """) + assertEquals(3, equalityFiles.size(), + "${tableName} must contain all three equality-delete files") + } finally { + sql """set enable_file_scanner_v2=true""" + sql """drop catalog if exists ${catalogName}""" + } +} diff --git a/regression-test/suites/external_table_p0/iceberg/test_iceberg_partition_evolution_filter_refs.groovy b/regression-test/suites/external_table_p0/iceberg/test_iceberg_partition_evolution_filter_refs.groovy new file mode 100644 index 00000000000000..bc31278706f414 --- /dev/null +++ b/regression-test/suites/external_table_p0/iceberg/test_iceberg_partition_evolution_filter_refs.groovy @@ -0,0 +1,275 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +suite("test_iceberg_partition_evolution_filter_refs", + "p0,external,iceberg,external_docker,external_docker_iceberg") { + String enabled = context.config.otherConfigs.get("enableIcebergTest") + if (enabled == null || !enabled.equalsIgnoreCase("true")) { + logger.info("disable iceberg test") + return + } + + String restPort = context.config.otherConfigs.get("iceberg_rest_uri_port") + String minioPort = context.config.otherConfigs.get("iceberg_minio_port") + String externalEnvIp = context.config.otherConfigs.get("externalEnvIp") + String catalogName = "test_iceberg_partition_evolution_filter_refs" + String dbName = "iceberg_partition_evolution_filter_refs_db" + String identityTable = "identity_bucket_truncate_timeline" + String temporalTable = "temporal_transform_timeline" + + def latestSnapshotId = { String tableName -> + List> rows = spark_iceberg """ + select snapshot_id + from demo.${dbName}.${tableName}.snapshots + order by committed_at desc + limit 1 + """ + assertEquals(1, rows.size()) + return rows[0][0].toString() + } + + sql """drop catalog if exists ${catalogName}""" + sql """ + create catalog ${catalogName} properties ( + 'type'='iceberg', + 'iceberg.catalog.type'='rest', + 'uri'='http://${externalEnvIp}:${restPort}', + 's3.access_key'='admin', + 's3.secret_key'='password', + 's3.endpoint'='http://${externalEnvIp}:${minioPort}', + 's3.region'='us-east-1', + 'meta.cache.iceberg.table.ttl-second'='0', + 'meta.cache.iceberg.schema.ttl-second'='0' + ) + """ + + try { + spark_iceberg_multi """ + create database if not exists demo.${dbName}; + drop table if exists demo.${dbName}.${identityTable}; + create table demo.${dbName}.${identityTable} ( + id int, + category string, + code string, + event_time timestamp, + payload struct + ) using iceberg + partitioned by (category, days(event_time)) + tblproperties ( + 'format-version'='2', + 'write.format.default'='parquet' + ); + insert into demo.${dbName}.${identityTable} values + (1, 'A', 'aa-1', timestamp '2026-01-01 01:00:00', + named_struct('metric', 10, 'label', 'base-a')), + (2, 'B', 'bb-1', timestamp '2026-01-02 01:00:00', + named_struct('metric', 20, 'label', 'base-b')); + """ + String identityBase = latestSnapshotId(identityTable) + sql """ + alter table `${catalogName}`.`${dbName}`.`${identityTable}` + create tag identity_base as of version ${identityBase} + """ + sql """ + alter table `${catalogName}`.`${dbName}`.`${identityTable}` + create branch identity_base_branch as of version ${identityBase} + """ + + // Scenario PE-I01: ADD bucket partition field and add a complex-type child in the same + // timeline. Filters must evaluate old files whose spec has no bucket field. + spark_iceberg_multi """ + alter table demo.${dbName}.${identityTable} add partition field bucket(8, id); + alter table demo.${dbName}.${identityTable} add column payload.extra string; + insert into demo.${dbName}.${identityTable} values + (3, 'A', 'aa-2', timestamp '2026-02-01 01:00:00', + named_struct('metric', 30, 'label', 'bucket-a', 'extra', 'add-child')), + (4, 'C', 'cc-1', timestamp '2026-02-02 01:00:00', + named_struct('metric', 40, 'label', 'bucket-c', 'extra', 'add-child')); + """ + String identityAdded = latestSnapshotId(identityTable) + sql """ + alter table `${catalogName}`.`${dbName}`.`${identityTable}` + create tag identity_added as of version ${identityAdded} + """ + + // Scenario PE-I02: REPLACE bucket with truncate while renaming/promoting nested children. + // Predicates on both old and new partition source columns must scan every applicable spec. + spark_iceberg_multi """ + alter table demo.${dbName}.${identityTable} + replace partition field bucket(8, id) with truncate(2, code); + alter table demo.${dbName}.${identityTable} + rename column payload.label to renamed_label; + alter table demo.${dbName}.${identityTable} + alter column payload.metric type bigint; + insert into demo.${dbName}.${identityTable} values + (5, 'A', 'aa-3', timestamp '2026-03-01 01:00:00', + named_struct('metric', 5000000000, 'renamed_label', 'truncate-a', + 'extra', 'renamed-child')), + (6, 'D', 'dd-1', timestamp '2026-03-02 01:00:00', + named_struct('metric', 60, 'renamed_label', 'truncate-d', + 'extra', 'renamed-child')); + """ + String identityReplaced = latestSnapshotId(identityTable) + + // Scenario PE-I03: DROP identity and temporal fields, then drop/re-add a nested name. + // New unpartitioned-by-category files and old identity-partitioned files coexist. + spark_iceberg_multi """ + alter table demo.${dbName}.${identityTable} drop partition field category; + alter table demo.${dbName}.${identityTable} drop partition field days(event_time); + alter table demo.${dbName}.${identityTable} drop column payload.extra; + alter table demo.${dbName}.${identityTable} add column payload.extra bigint; + insert into demo.${dbName}.${identityTable} values + (7, 'A', 'aa-4', timestamp '2026-04-01 01:00:00', + named_struct('metric', 70, 'renamed_label', 'dropped-partition', + 'extra', 7000)), + (8, 'E', 'ee-1', timestamp '2026-04-02 01:00:00', + named_struct('metric', 80, 'renamed_label', 'dropped-partition', + 'extra', 8000)); + """ + String identityDropped = latestSnapshotId(identityTable) + sql """ + alter table `${catalogName}`.`${dbName}`.`${identityTable}` + create tag identity_dropped as of version ${identityDropped} + """ + + spark_iceberg_multi """ + drop table if exists demo.${dbName}.${temporalTable}; + create table demo.${dbName}.${temporalTable} ( + id int, + event_time timestamp, + payload string + ) using iceberg + partitioned by (years(event_time)) + tblproperties ( + 'format-version'='2', + 'write.format.default'='orc' + ); + insert into demo.${dbName}.${temporalTable} values + (11, timestamp '2024-01-01 01:00:00', 'year-2024'), + (12, timestamp '2025-01-01 01:00:00', 'year-2025'); + """ + String temporalYear = latestSnapshotId(temporalTable) + sql """ + alter table `${catalogName}`.`${dbName}`.`${temporalTable}` + create tag temporal_year as of version ${temporalYear} + """ + + // Scenario PE-T01: REPLACE year -> month -> day -> hour across ORC files. + // Range and equality filters validate every temporal transform boundary. + spark_iceberg_multi """ + alter table demo.${dbName}.${temporalTable} + replace partition field years(event_time) with months(event_time); + insert into demo.${dbName}.${temporalTable} values + (13, timestamp '2026-02-01 01:00:00', 'month-feb'), + (14, timestamp '2026-03-01 01:00:00', 'month-mar'); + alter table demo.${dbName}.${temporalTable} + replace partition field months(event_time) with days(event_time); + insert into demo.${dbName}.${temporalTable} values + (15, timestamp '2026-04-03 01:00:00', 'day-03'), + (16, timestamp '2026-04-04 01:00:00', 'day-04'); + alter table demo.${dbName}.${temporalTable} + replace partition field days(event_time) with hours(event_time); + insert into demo.${dbName}.${temporalTable} values + (17, timestamp '2026-05-01 08:00:00', 'hour-08'), + (18, timestamp '2026-05-01 09:00:00', 'hour-09'); + """ + String temporalHour = latestSnapshotId(temporalTable) + sql """ + alter table `${catalogName}`.`${dbName}`.`${temporalTable}` + create tag temporal_hour as of version ${temporalHour} + """ + + sql """switch ${catalogName}""" + sql """use ${dbName}""" + sql """refresh catalog ${catalogName}""" + + // Scenario PE-F01: equality/range/IN/NULL-safe source-column filters span four specs. + qt_current_identity_filter """ + select id from ${identityTable} where category = 'A' order by id + """ + qt_current_truncate_source_filter """ + select id from ${identityTable} + where code in ('aa-1', 'aa-2', 'aa-3', 'aa-4') + order by id + """ + qt_current_temporal_source_filter """ + select id from ${identityTable} + where event_time >= timestamp '2026-03-01 00:00:00' + order by id + """ + qt_current_readded_nested_field """ + select id, payload.extra from ${identityTable} + where payload.extra is not null + order by id + """ + + // Scenario PE-R01: numeric snapshot, tag and branch retain their own data/spec timeline. + qt_identity_base_snapshot """ + select id, category + from ${identityTable} for version as of ${identityBase} + where category in ('A', 'B') + order by id + """ + qt_identity_base_tag """ + select id, category from ${identityTable}@tag(identity_base) + where category in ('A', 'B') order by id + """ + qt_identity_base_branch """ + select id, category from ${identityTable}@branch(identity_base_branch) + where category in ('A', 'B') order by id + """ + qt_identity_added_snapshot """ + select id from ${identityTable} for version as of ${identityAdded} + where category = 'A' order by id + """ + qt_identity_replaced_snapshot """ + select id from ${identityTable} for version as of ${identityReplaced} + where category = 'A' order by id + """ + qt_identity_dropped_tag """ + select id from ${identityTable}@tag(identity_dropped) + where category = 'A' order by id + """ + + // Scenario PE-F02/PE-R02: temporal filters use the spec selected by numeric/tag refs. + qt_temporal_year_tag """ + select id from ${temporalTable}@tag(temporal_year) + where event_time < timestamp '2026-01-01 00:00:00' + order by id + """ + // This current predicate deliberately crosses files written by year, month, day and hour + // specs so losing any older transform boundary changes the fixed result contract. + qt_temporal_current_all_specs """ + select id from ${temporalTable} + where event_time >= timestamp '2024-01-01 00:00:00' + and event_time < timestamp '2026-05-02 00:00:00' + order by id + """ + qt_temporal_hour_snapshot """ + select id from ${temporalTable} for version as of ${temporalHour} + where event_time >= timestamp '2026-05-01 00:00:00' + and event_time < timestamp '2026-05-02 00:00:00' + order by id + """ + qt_temporal_hour_tag """ + select id from ${temporalTable}@tag(temporal_hour) + where event_time = timestamp '2026-05-01 09:00:00' + """ + } finally { + sql """drop catalog if exists ${catalogName}""" + } +} diff --git a/regression-test/suites/external_table_p0/iceberg/test_iceberg_partition_evolution_format_scanner.groovy b/regression-test/suites/external_table_p0/iceberg/test_iceberg_partition_evolution_format_scanner.groovy new file mode 100644 index 00000000000000..7b492b67f0e192 --- /dev/null +++ b/regression-test/suites/external_table_p0/iceberg/test_iceberg_partition_evolution_format_scanner.groovy @@ -0,0 +1,211 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +suite("test_iceberg_partition_evolution_format_scanner", + "p0,external,iceberg,external_docker,external_docker_iceberg") { + String enabled = context.config.otherConfigs.get("enableIcebergTest") + if (enabled == null || !enabled.equalsIgnoreCase("true")) { + logger.info("disable iceberg test") + return + } + + String restPort = context.config.otherConfigs.get("iceberg_rest_uri_port") + String minioPort = context.config.otherConfigs.get("iceberg_minio_port") + String externalEnvIp = context.config.otherConfigs.get("externalEnvIp") + String catalogName = "test_iceberg_partition_evolution_format_scanner" + String dbName = "iceberg_partition_evolution_format_scanner_db" + + def latestSnapshotId = { String tableName -> + return spark_iceberg(""" + select snapshot_id + from demo.${dbName}.${tableName}.snapshots + order by committed_at desc + limit 1 + """)[0][0].toString() + } + + sql """drop catalog if exists ${catalogName}""" + sql """ + create catalog ${catalogName} properties ( + 'type'='iceberg', + 'iceberg.catalog.type'='rest', + 'uri'='http://${externalEnvIp}:${restPort}', + 's3.access_key'='admin', + 's3.secret_key'='password', + 's3.endpoint'='http://${externalEnvIp}:${minioPort}', + 's3.region'='us-east-1', + 'meta.cache.iceberg.table.ttl-second'='0', + 'meta.cache.iceberg.schema.ttl-second'='0' + ) + """ + + try { + ["parquet", "orc"].each { String format -> + String identityTable = "identity_bucket_truncate_${format}" + String temporalTable = "temporal_transform_${format}" + + // Scenario PE-X01: run the complete identity + bucket -> truncate + drop timeline + // for each data format, including a nested add/rename/promotion/drop-readd sequence. + spark_iceberg_multi """ + create database if not exists demo.${dbName}; + drop table if exists demo.${dbName}.${identityTable}; + create table demo.${dbName}.${identityTable} ( + id int, + category string, + code string, + event_time timestamp, + payload struct + ) using iceberg + partitioned by (category, days(event_time)) + tblproperties ( + 'format-version'='2', + 'write.format.default'='${format}' + ); + insert into demo.${dbName}.${identityTable} values + (1, 'A', 'aa-1', timestamp '2026-01-01 01:00:00', + named_struct('metric', 10, 'label', 'base-a')), + (2, 'B', 'bb-1', timestamp '2026-01-02 01:00:00', + named_struct('metric', 20, 'label', 'base-b')); + """ + String identityBase = latestSnapshotId(identityTable) + sql """ + alter table `${catalogName}`.`${dbName}`.`${identityTable}` + create tag ${identityTable}_base as of version ${identityBase} + """ + spark_iceberg_multi """ + alter table demo.${dbName}.${identityTable} add partition field bucket(8, id); + alter table demo.${dbName}.${identityTable} add column payload.extra string; + insert into demo.${dbName}.${identityTable} values + (3, 'A', 'aa-2', timestamp '2026-02-01 01:00:00', + named_struct('metric', 30, 'label', 'bucket-a', 'extra', 'added')), + (4, 'C', 'cc-1', timestamp '2026-02-02 01:00:00', + named_struct('metric', 40, 'label', 'bucket-c', 'extra', 'added')); + alter table demo.${dbName}.${identityTable} + replace partition field bucket(8, id) with truncate(2, code); + alter table demo.${dbName}.${identityTable} + rename column payload.label to renamed_label; + alter table demo.${dbName}.${identityTable} + alter column payload.metric type bigint; + insert into demo.${dbName}.${identityTable} values + (5, 'A', 'aa-3', timestamp '2026-03-01 01:00:00', + named_struct('metric', 5000000000, 'renamed_label', 'truncate-a', + 'extra', 'renamed')), + (6, 'D', 'dd-1', timestamp '2026-03-02 01:00:00', + named_struct('metric', 60, 'renamed_label', 'truncate-d', + 'extra', 'renamed')); + alter table demo.${dbName}.${identityTable} drop partition field category; + alter table demo.${dbName}.${identityTable} drop partition field days(event_time); + alter table demo.${dbName}.${identityTable} drop column payload.extra; + alter table demo.${dbName}.${identityTable} add column payload.extra bigint; + insert into demo.${dbName}.${identityTable} values + (7, 'A', 'aa-4', timestamp '2026-04-01 01:00:00', + named_struct('metric', 70, 'renamed_label', 'drop-a', 'extra', 7000)), + (8, 'E', 'ee-1', timestamp '2026-04-02 01:00:00', + named_struct('metric', 80, 'renamed_label', 'drop-e', 'extra', 8000)); + """ + String identityFinal = latestSnapshotId(identityTable) + sql """ + alter table `${catalogName}`.`${dbName}`.`${identityTable}` + create tag ${identityTable}_final as of version ${identityFinal} + """ + + // Scenario PE-X02: run the complete year -> month -> day -> hour replacement timeline + // in the same format so each transform boundary is observable under both scanners. + spark_iceberg_multi """ + drop table if exists demo.${dbName}.${temporalTable}; + create table demo.${dbName}.${temporalTable} ( + id int, + event_time timestamp, + payload string + ) using iceberg + partitioned by (years(event_time)) + tblproperties ( + 'format-version'='2', + 'write.format.default'='${format}' + ); + insert into demo.${dbName}.${temporalTable} values + (11, timestamp '2024-01-01 01:00:00', 'year-2024'), + (12, timestamp '2025-01-01 01:00:00', 'year-2025'); + alter table demo.${dbName}.${temporalTable} + replace partition field years(event_time) with months(event_time); + insert into demo.${dbName}.${temporalTable} values + (13, timestamp '2026-02-01 01:00:00', 'month-feb'), + (14, timestamp '2026-03-01 01:00:00', 'month-mar'); + alter table demo.${dbName}.${temporalTable} + replace partition field months(event_time) with days(event_time); + insert into demo.${dbName}.${temporalTable} values + (15, timestamp '2026-04-03 01:00:00', 'day-03'), + (16, timestamp '2026-04-04 01:00:00', 'day-04'); + alter table demo.${dbName}.${temporalTable} + replace partition field days(event_time) with hours(event_time); + insert into demo.${dbName}.${temporalTable} values + (17, timestamp '2026-05-01 08:00:00', 'hour-08'), + (18, timestamp '2026-05-01 09:00:00', 'hour-09'); + """ + String temporalFinal = latestSnapshotId(temporalTable) + sql """ + alter table `${catalogName}`.`${dbName}`.`${temporalTable}` + create tag ${temporalTable}_final as of version ${temporalFinal} + """ + + sql """switch ${catalogName}""" + sql """use ${dbName}""" + sql """refresh catalog ${catalogName}""" + + [true, false].each { boolean scannerV2 -> + String scanner = scannerV2 ? "v2" : "v1" + sql """set enable_file_scanner_v2=${scannerV2}""" + + // Scenario PE-X03: every format/scanner cell reads current data spanning every + // identity/bucket/truncate/drop spec and preserves nested field IDs. + "qt_identity_${format}_${scanner}_current"(""" + select id, payload.extra from ${identityTable} + where category = 'A' or code in ('bb-1', 'cc-1', 'dd-1', 'ee-1') + order by id + """) + "qt_identity_${format}_${scanner}_base_tag"(""" + select id, category, payload.label + from ${identityTable}@tag(${identityTable}_base) + order by id + """) + "qt_identity_${format}_${scanner}_final_tag"(""" + select id, payload.renamed_label + from ${identityTable}@tag(${identityTable}_final) + where category = 'A' + order by id + """) + + // Scenario PE-X04: the current predicate spans year/month/day/hour files and the + // historical tag is read by every format/scanner combination. + "qt_temporal_${format}_${scanner}_current"(""" + select id from ${temporalTable} + where event_time >= timestamp '2024-01-01 00:00:00' + and event_time < timestamp '2026-05-02 00:00:00' + order by id + """) + "qt_temporal_${format}_${scanner}_final_tag"(""" + select id from ${temporalTable}@tag(${temporalTable}_final) + where event_time >= timestamp '2026-02-01 00:00:00' + order by id + """) + } + } + } finally { + sql """set enable_file_scanner_v2=true""" + sql """drop catalog if exists ${catalogName}""" + } +} diff --git a/regression-test/suites/external_table_p0/iceberg/test_iceberg_partition_evolution_position_dv.groovy b/regression-test/suites/external_table_p0/iceberg/test_iceberg_partition_evolution_position_dv.groovy new file mode 100644 index 00000000000000..4ed60374d04a3d --- /dev/null +++ b/regression-test/suites/external_table_p0/iceberg/test_iceberg_partition_evolution_position_dv.groovy @@ -0,0 +1,318 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +import org.apache.doris.regression.action.ProfileAction + +suite("test_iceberg_partition_evolution_position_dv", + "p0,external,iceberg,external_docker,external_docker_iceberg,nonConcurrent") { + String enabled = context.config.otherConfigs.get("enableIcebergTest") + if (enabled == null || !enabled.equalsIgnoreCase("true")) { + logger.info("disable iceberg test") + return + } + + String restPort = context.config.otherConfigs.get("iceberg_rest_uri_port") + String minioPort = context.config.otherConfigs.get("iceberg_minio_port") + String externalEnvIp = context.config.otherConfigs.get("externalEnvIp") + String catalogName = "test_iceberg_partition_evolution_position_dv" + String dbName = "iceberg_partition_evolution_position_dv_db" + + def stringRows = { String query -> + sql(query).collect { row -> + row.collect { value -> value == null ? null : value.toString() } + } + } + def latestSnapshotId = { String tableName -> + return spark_iceberg(""" + select snapshot_id + from demo.${dbName}.${tableName}.snapshots + order by committed_at desc + limit 1 + """)[0][0].toString() + } + def profileAction = new ProfileAction(context) + def profileCounterValues = { String profileText, String counterName -> + def values = [] + def matcher = profileText =~ ("(?m)^\\s*(?:-\\s*)?" + + java.util.regex.Pattern.quote(counterName) + ":\\s+([^\\n]+)") + while (matcher.find()) { + String valueText = matcher.group(1).toString() + def exact = valueText =~ /\(([0-9,]+)\)/ + def number = valueText =~ /([0-9,]+)/ + String rawValue = exact.find() ? exact.group(1) : (number.find() ? number.group(1) : null) + if (rawValue != null) { + values.add(Long.parseLong(rawValue.replace(",", ""))) + } + } + return values + } + def assertRuntimeFilterPruned = { String tableName, String dimensionTable -> + String token = UUID.randomUUID().toString() + List> rows = stringRows(""" + select /*+ SET_VAR(runtime_filter_type='IN_OR_BLOOM_FILTER') */ + '${token}', f.id + from ${tableName} f + join ${dimensionTable} d on f.category = d.category + order by f.id + """) + String profile = profileAction.getProfileBySql( + token, + ["RuntimeFilterPartitionPrunedRangeNum"], + 30000L, + 500L) + long fileRangesPruned = profileCounterValues( + profile, "RuntimeFilterPartitionPrunedRangeNum").sum(0L) + long partitionsPruned = profileCounterValues( + profile, "PartitionsPrunedByRuntimeFilter").sum(0L) + assertTrue(fileRangesPruned + partitionsPruned > 0L, + "Runtime filter did not prune a delete-aware Iceberg partition/file range; " + + profile.take(2000).replaceAll("\\s+", " ")) + return rows.collect { row -> [row[1]] } + } + + sql """drop catalog if exists ${catalogName}""" + sql """ + create catalog ${catalogName} properties ( + 'type'='iceberg', + 'iceberg.catalog.type'='rest', + 'uri'='http://${externalEnvIp}:${restPort}', + 's3.access_key'='admin', + 's3.secret_key'='password', + 's3.endpoint'='http://${externalEnvIp}:${minioPort}', + 's3.region'='us-east-1', + 'meta.cache.iceberg.table.ttl-second'='0' + ) + """ + + try { + ["parquet", "orc"].each { String format -> + [2, 3].each { int formatVersion -> + String deleteKind = formatVersion == 2 ? "position" : "dv" + String tableName = "${deleteKind}_${format}_evolved" + + spark_iceberg_multi """ + create database if not exists demo.${dbName}; + drop table if exists demo.${dbName}.${tableName}; + create table demo.${dbName}.${tableName} ( + id int, + category string, + code string, + event_time timestamp, + payload struct + ) using iceberg + partitioned by (category, days(event_time)) + tblproperties ( + 'format-version'='${formatVersion}', + 'write.format.default'='${format}', + 'write.delete.mode'='merge-on-read', + 'write.update.mode'='merge-on-read', + 'write.merge.mode'='merge-on-read', + 'write.distribution-mode'='none' + ); + insert into demo.${dbName}.${tableName} + select /*+ coalesce(1) */ id, category, code, event_time, payload from values + (1, 'A', 'aa-1', timestamp '2026-01-01 01:00:00', + named_struct('metric', 10, 'label', 'base-a')), + (2, 'A', 'aa-2', timestamp '2026-01-01 02:00:00', + named_struct('metric', 20, 'label', 'base-delete')), + (3, 'B', 'bb-1', timestamp '2026-01-02 01:00:00', + named_struct('metric', 30, 'label', 'base-b')), + (4, 'C', 'cc-1', timestamp '2026-01-03 01:00:00', + named_struct('metric', 40, 'label', 'base-c')) + as t(id, category, code, event_time, payload); + """ + String baseSnapshot = latestSnapshotId(tableName) + sql """ + alter table `${catalogName}`.`${dbName}`.`${tableName}` + create tag ${tableName}_base as of version ${baseSnapshot} + """ + + // Scenario PE-D01: delete against the original spec before any evolution. + spark_iceberg """ + delete from demo.${dbName}.${tableName} where id = 2 + """ + String firstDeleteSnapshot = latestSnapshotId(tableName) + + // Scenario PE-D02: ADD partition field and complex child, then delete a row written + // with the new spec. Old/new delete files must remain associated with their specs. + spark_iceberg_multi """ + alter table demo.${dbName}.${tableName} add partition field bucket(8, id); + alter table demo.${dbName}.${tableName} add column payload.extra string; + insert into demo.${dbName}.${tableName} + select /*+ coalesce(1) */ id, category, code, event_time, payload from values + (5, 'A', 'aa-3', timestamp '2026-02-01 01:00:00', + named_struct('metric', 50, 'label', 'add-a', 'extra', 'new-child')), + (6, 'A', 'aa-4', timestamp '2026-02-01 02:00:00', + named_struct('metric', 60, 'label', 'add-delete', 'extra', 'new-child')) + as t(id, category, code, event_time, payload); + delete from demo.${dbName}.${tableName} where id = 6; + """ + String addedDeleteSnapshot = latestSnapshotId(tableName) + + // Scenario PE-D03: REPLACE temporal transform and rename/promote nested fields. + spark_iceberg_multi """ + alter table demo.${dbName}.${tableName} + replace partition field days(event_time) with months(event_time); + alter table demo.${dbName}.${tableName} + replace partition field bucket(8, id) with truncate(2, code); + alter table demo.${dbName}.${tableName} + rename column payload.label to renamed_label; + alter table demo.${dbName}.${tableName} + alter column payload.metric type bigint; + insert into demo.${dbName}.${tableName} values + (7, 'A', 'aa-5', timestamp '2026-03-01 01:00:00', + named_struct('metric', 7000000000, + 'renamed_label', 'replace-a', 'extra', 'renamed-child')), + (8, 'A', 'aa-6', timestamp '2026-03-01 02:00:00', + named_struct('metric', 80, + 'renamed_label', 'replace-delete', 'extra', 'renamed-child')); + """ + + // Scenario PE-D04: DROP identity field, write a victim with the resulting spec, + // then delete that victim. The data and delete artifacts must both use the spec + // that lacks category while older files still expose category partitions. + spark_iceberg_multi """ + alter table demo.${dbName}.${tableName} drop partition field category; + delete from demo.${dbName}.${tableName} where id = 8; + insert into demo.${dbName}.${tableName} + select /*+ coalesce(1) */ id, category, code, event_time, payload from values + (9, 'A', 'aa-7', timestamp '2026-04-01 01:00:00', + named_struct('metric', 90, + 'renamed_label', 'drop-delete', 'extra', 'new-spec')), + (10, 'A', 'aa-8', timestamp '2026-04-01 02:00:00', + named_struct('metric', 100, + 'renamed_label', 'drop-survivor', 'extra', 'new-spec')) + as t(id, category, code, event_time, payload); + delete from demo.${dbName}.${tableName} where id = 9; + drop table if exists demo.${dbName}.${tableName}_dimension; + create table demo.${dbName}.${tableName}_dimension (category string) + using iceberg tblproperties ('format-version'='2'); + insert into demo.${dbName}.${tableName}_dimension values ('A'); + """ + List> dataSpecRows = spark_iceberg """ + select max(spec_id) + from demo.${dbName}.${tableName}.all_files + where content = 0 + """ + List> deleteSpecRows = spark_iceberg """ + select max(spec_id) + from demo.${dbName}.${tableName}.all_files + where content = 1 + """ + assertEquals(dataSpecRows[0][0], deleteSpecRows[0][0], + "The post-drop victim data and delete artifacts must use the same spec") + String finalSnapshot = latestSnapshotId(tableName) + sql """ + alter table `${catalogName}`.`${dbName}`.`${tableName}` + create tag ${tableName}_final as of version ${finalSnapshot} + """ + + sql """switch ${catalogName}""" + sql """use ${dbName}""" + sql """refresh table ${tableName}""" + String actionSuffix = "${deleteKind}_${format}" + + // Scenario PE-D05: static partition filters apply every delete exactly once. + List> expectedCurrent = [["1"], ["5"], ["7"], ["10"]] + "qt_${actionSuffix}_current_filter"(""" + select id from ${tableName} where category = 'A' order by id + """) + "qt_${actionSuffix}_current_nested"(""" + select id, payload.metric from ${tableName} + where payload.metric > 5000000000 order by id + """) + + // Scenario PE-D06: runtime filter on the dropped identity partition column returns + // the same delete-aware rows with pruning disabled, then proves that the enabled + // query retains an RF and prunes at least one partition/file range. + String rfQuery = """ + select /*+ SET_VAR(runtime_filter_type='IN_OR_BLOOM_FILTER') */ f.id + from ${tableName} f + join ${tableName}_dimension d on f.category = d.category + order by f.id + """ + sql """set runtime_filter_wait_infinitely=true""" + sql """set disable_join_reorder=true""" + sql """set enable_runtime_filter_prune=false""" + sql """set runtime_filter_mode=GLOBAL""" + sql """set parallel_pipeline_task_num=1""" + sql """set enable_profile=true""" + sql """set profile_level=2""" + sql """set enable_runtime_filter_partition_prune=false""" + "qt_${actionSuffix}_rf_disabled"(rfQuery) + sql """set enable_runtime_filter_partition_prune=true""" + assertEquals(expectedCurrent, + assertRuntimeFilterPruned(tableName, "${tableName}_dimension")) + + // Scenario PE-D07: numeric snapshots and tags preserve historical rows and deletes. + "qt_${actionSuffix}_base_snapshot"(""" + select id from ${tableName} for version as of ${baseSnapshot} + where category in ('A', 'B', 'C') order by id + """) + "qt_${actionSuffix}_base_tag"(""" + select id from ${tableName}@tag(${tableName}_base) + where category = 'A' order by id + """) + "qt_${actionSuffix}_first_delete_snapshot"(""" + select id from ${tableName} for version as of ${firstDeleteSnapshot} + where category = 'A' order by id + """) + "qt_${actionSuffix}_added_delete_snapshot"(""" + select id from ${tableName} for version as of ${addedDeleteSnapshot} + where category = 'A' order by id + """) + "qt_${actionSuffix}_final_tag"(""" + select id from ${tableName}@tag(${tableName}_final) + where category = 'A' order by id + """) + + // Scenario PE-D08: legacy and V2 scanners must agree on multi-spec delete planning. + sql """set enable_file_scanner_v2=true""" + "qt_${actionSuffix}_scanner_v2"(""" + select id from ${tableName} where category = 'A' order by id + """) + sql """set enable_file_scanner_v2=false""" + "qt_${actionSuffix}_scanner_v1"(""" + select id from ${tableName} where category = 'A' order by id + """) + sql """set enable_file_scanner_v2=true""" + + // Scenario PE-D09: verify the fixture really contains physical delete artifacts. + // Iceberg may replace or consolidate older artifacts across delete commits, while + // the snapshot assertions above still validate every pre/post-evolution phase. + List> deleteFiles = stringRows(""" + select file_format + from ${tableName}\$all_files + where content = 1 + order by file_format + """) + assertTrue(deleteFiles.size() >= 1, + "${tableName} must contain at least one physical delete artifact") + if (formatVersion == 3) { + assertTrue(deleteFiles.any { row -> row[0].equalsIgnoreCase("PUFFIN") }, + "${tableName} must contain Iceberg deletion vectors") + } + } + } + } finally { + sql """set enable_file_scanner_v2=true""" + sql """set enable_runtime_filter_prune=true""" + sql """set enable_runtime_filter_partition_prune=true""" + sql """set disable_join_reorder=false""" + sql """drop catalog if exists ${catalogName}""" + } +} diff --git a/regression-test/suites/external_table_p0/iceberg/test_iceberg_partition_evolution_runtime_filter.groovy b/regression-test/suites/external_table_p0/iceberg/test_iceberg_partition_evolution_runtime_filter.groovy new file mode 100644 index 00000000000000..729ba0418b5ff0 --- /dev/null +++ b/regression-test/suites/external_table_p0/iceberg/test_iceberg_partition_evolution_runtime_filter.groovy @@ -0,0 +1,277 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +import org.apache.doris.regression.action.ProfileAction + +suite("test_iceberg_partition_evolution_runtime_filter", + "p0,external,iceberg,external_docker,external_docker_iceberg,nonConcurrent") { + String enabled = context.config.otherConfigs.get("enableIcebergTest") + if (enabled == null || !enabled.equalsIgnoreCase("true")) { + logger.info("disable iceberg test") + return + } + + String restPort = context.config.otherConfigs.get("iceberg_rest_uri_port") + String minioPort = context.config.otherConfigs.get("iceberg_minio_port") + String externalEnvIp = context.config.otherConfigs.get("externalEnvIp") + String catalogName = "test_iceberg_partition_evolution_runtime_filter" + String dbName = "iceberg_partition_evolution_runtime_filter_db" + String factTable = "evolved_fact" + String dimensionTable = "rf_dimension" + + def stringRows = { String query -> + sql(query).collect { row -> + row.collect { value -> value == null ? null : value.toString() } + } + } + def latestSnapshotId = { + return spark_iceberg(""" + select snapshot_id + from demo.${dbName}.${factTable}.snapshots + order by committed_at desc + limit 1 + """)[0][0].toString() + } + def profileAction = new ProfileAction(context) + def profileCounterValues = { String profileText, String counterName -> + def values = [] + def matcher = profileText =~ ("(?m)^\\s*(?:-\\s*)?" + + java.util.regex.Pattern.quote(counterName) + ":\\s+([^\\n]+)") + while (matcher.find()) { + String valueText = matcher.group(1).toString() + def exact = valueText =~ /\(([0-9,]+)\)/ + def number = valueText =~ /([0-9,]+)/ + String rawValue = exact.find() ? exact.group(1) : (number.find() ? number.group(1) : null) + if (rawValue != null) { + values.add(Long.parseLong(rawValue.replace(",", ""))) + } + } + return values + } + def assertRuntimeFilterPruned = { String queryBody -> + String token = UUID.randomUUID().toString() + List> rows = stringRows(""" + select /*+ SET_VAR(runtime_filter_type='IN_OR_BLOOM_FILTER') */ + '${token}', f.id + ${queryBody} + order by f.id + """) + // Scanner counters can arrive after the profile list first reports COMPLETE. + String profile = profileAction.getProfileBySql( + token, + ["RuntimeFilterPartitionPrunedRangeNum"], + 30000L, + 500L) + long fileRangesPruned = profileCounterValues( + profile, "RuntimeFilterPartitionPrunedRangeNum").sum(0L) + long partitionsPruned = profileCounterValues( + profile, "PartitionsPrunedByRuntimeFilter").sum(0L) + assertTrue(fileRangesPruned + partitionsPruned > 0L, + "Runtime filter did not prune any evolved Iceberg partition/file range; " + + profile.take(2000).replaceAll("\\s+", " ")) + return rows.collect { row -> [row[1]] } + } + + sql """drop catalog if exists ${catalogName}""" + sql """ + create catalog ${catalogName} properties ( + 'type'='iceberg', + 'iceberg.catalog.type'='rest', + 'uri'='http://${externalEnvIp}:${restPort}', + 's3.access_key'='admin', + 's3.secret_key'='password', + 's3.endpoint'='http://${externalEnvIp}:${minioPort}', + 's3.region'='us-east-1', + 'meta.cache.iceberg.table.ttl-second'='0' + ) + """ + + try { + spark_iceberg_multi """ + create database if not exists demo.${dbName}; + drop table if exists demo.${dbName}.${factTable}; + create table demo.${dbName}.${factTable} ( + id int, + category string, + region string, + event_time timestamp, + payload struct + ) using iceberg + partitioned by (category, days(event_time)) + tblproperties ('format-version'='2'); + insert into demo.${dbName}.${factTable} values + (1, 'A', 'r1', timestamp '2026-01-01 01:00:00', + named_struct('metric', 10)), + (2, 'B', 'r2', timestamp '2026-01-02 01:00:00', + named_struct('metric', 20)), + (3, 'C', 'r3', timestamp '2026-01-03 01:00:00', + named_struct('metric', 30)); + """ + String baseSnapshot = latestSnapshotId() + sql """ + alter table `${catalogName}`.`${dbName}`.`${factTable}` + create tag rf_base as of version ${baseSnapshot} + """ + + // Scenario PE-RF01: add identity and bucket fields and evolve a nested payload between + // data files. Old files lack both partition values but still contain the source columns. + spark_iceberg_multi """ + alter table demo.${dbName}.${factTable} add partition field region; + alter table demo.${dbName}.${factTable} add partition field bucket(8, id); + alter table demo.${dbName}.${factTable} add column payload.label string; + insert into demo.${dbName}.${factTable} values + (4, 'A', 'r1', timestamp '2026-02-01 01:00:00', + named_struct('metric', 40, 'label', 'add-spec')), + (5, 'D', 'r4', timestamp '2026-02-02 01:00:00', + named_struct('metric', 50, 'label', 'add-spec')); + """ + String addedSnapshot = latestSnapshotId() + + // Scenario PE-RF02: replace a temporal transform. Runtime filters on event_time must + // translate against the transform of each file's own spec. + spark_iceberg_multi """ + alter table demo.${dbName}.${factTable} + replace partition field days(event_time) with months(event_time); + insert into demo.${dbName}.${factTable} values + (6, 'A', 'r1', timestamp '2026-03-01 01:00:00', + named_struct('metric', 60, 'label', 'replace-spec')), + (7, 'E', 'r5', timestamp '2026-03-02 01:00:00', + named_struct('metric', 70, 'label', 'replace-spec')); + """ + + // Scenario PE-RF03: drop the identity field. New files have no category partition value; + // runtime pruning may only discard older ranges and must still scan matching new rows. + spark_iceberg_multi """ + alter table demo.${dbName}.${factTable} drop partition field category; + insert into demo.${dbName}.${factTable} values + (8, 'A', 'r1', timestamp '2026-04-01 01:00:00', + named_struct('metric', 80, 'label', 'drop-spec')), + (9, 'F', 'r6', timestamp '2026-04-02 01:00:00', + named_struct('metric', 90, 'label', 'drop-spec')); + drop table if exists demo.${dbName}.${dimensionTable}; + create table demo.${dbName}.${dimensionTable} ( + category string, + region string, + id int, + lower_time timestamp, + upper_time timestamp + ) using iceberg + tblproperties ('format-version'='2'); + insert into demo.${dbName}.${dimensionTable} values + ('A', 'r1', 4, + timestamp '2026-03-01 00:00:00', timestamp '2026-05-01 00:00:00'); + """ + String droppedSnapshot = latestSnapshotId() + sql """ + alter table `${catalogName}`.`${dbName}`.`${factTable}` + create tag rf_dropped as of version ${droppedSnapshot} + """ + + sql """switch ${catalogName}""" + sql """use ${dbName}""" + sql """refresh catalog ${catalogName}""" + sql """set enable_profile=true""" + sql """set profile_level=2""" + // Small fixture tables have no column statistics. Keep the generated RF so this suite + // validates scanner-side partition pruning instead of optimizer selectivity heuristics. + sql """set enable_runtime_filter_prune=false""" + sql """set runtime_filter_wait_infinitely=true""" + sql """set runtime_filter_mode=GLOBAL""" + sql """set parallel_pipeline_task_num=1""" + sql """set disable_join_reorder=true""" + + String currentCategoryJoin = """ + from ${factTable} f + join ${dimensionTable} d on f.category = d.category + """ + String addedIdentityJoin = """ + from ${factTable} f + join ${dimensionTable} d on f.region = d.region + """ + String bucketSourceJoin = """ + from ${factTable} f + join ${dimensionTable} d on f.id = d.id + """ + String currentTemporalJoin = """ + from ${factTable} f + join ${dimensionTable} d + on f.event_time >= d.lower_time and f.event_time < d.upper_time + """ + + // Scenario PE-RF04: result parity with RF disabled protects correctness. + sql """set enable_runtime_filter_partition_prune=false""" + qt_category_rf_disabled """ + select f.id ${currentCategoryJoin} order by f.id + """ + qt_added_identity_rf_disabled """ + select f.id ${addedIdentityJoin} order by f.id + """ + qt_bucket_source_rf_disabled """ + select f.id ${bucketSourceJoin} order by f.id + """ + qt_temporal_rf_disabled """ + select f.id ${currentTemporalJoin} order by f.id + """ + + // Scenario PE-RF05: independently profile the original identity field that is later + // dropped and the identity field added after old files were written. + sql """set enable_runtime_filter_partition_prune=true""" + assertEquals([["1"], ["4"], ["6"], ["8"]], + assertRuntimeFilterPruned(currentCategoryJoin)) + assertEquals([["1"], ["4"], ["6"], ["8"]], + assertRuntimeFilterPruned(addedIdentityJoin)) + + // Scenario PE-RF06: bucket and temporal source-column RFs retain result correctness. + // Scanner-side RF pruning currently consumes identity partition values only, so these + // cells intentionally do not claim a positive physical-pruning counter. + qt_bucket_source_rf_enabled """ + select /*+ SET_VAR(runtime_filter_type='IN_OR_BLOOM_FILTER') */ f.id + ${bucketSourceJoin} + order by f.id + """ + qt_temporal_rf_enabled """ + select /*+ SET_VAR(runtime_filter_type='IN_OR_BLOOM_FILTER') */ f.id + ${currentTemporalJoin} + order by f.id + """ + + // Scenario PE-RF07: numeric snapshot and tag retain runtime-filter correctness. + qt_base_snapshot_rf """ + select /*+ SET_VAR(runtime_filter_type='IN_OR_BLOOM_FILTER') */ f.id + from ${factTable} for version as of ${baseSnapshot} f + join ${dimensionTable} d on f.category = d.category + order by f.id + """ + qt_added_snapshot_rf """ + select /*+ SET_VAR(runtime_filter_type='IN_OR_BLOOM_FILTER') */ f.id + from ${factTable} for version as of ${addedSnapshot} f + join ${dimensionTable} d on f.region = d.region + order by f.id + """ + qt_dropped_tag_rf """ + select /*+ SET_VAR(runtime_filter_type='IN_OR_BLOOM_FILTER') */ f.id + from ${factTable}@tag(rf_dropped) f + join ${dimensionTable} d on f.category = d.category + order by f.id + """ + } finally { + sql """set enable_runtime_filter_prune=true""" + sql """set enable_runtime_filter_partition_prune=true""" + sql """set disable_join_reorder=false""" + sql """drop catalog if exists ${catalogName}""" + } +} diff --git a/regression-test/suites/external_table_p0/iceberg/test_iceberg_schema_dual_relation_matrix.groovy b/regression-test/suites/external_table_p0/iceberg/test_iceberg_schema_dual_relation_matrix.groovy new file mode 100644 index 00000000000000..a9fcfb4aeb7fa7 --- /dev/null +++ b/regression-test/suites/external_table_p0/iceberg/test_iceberg_schema_dual_relation_matrix.groovy @@ -0,0 +1,191 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +suite("test_iceberg_schema_dual_relation_matrix", + "p0,external,iceberg,external_docker,external_docker_iceberg") { + String enabled = context.config.otherConfigs.get("enableIcebergTest") + if (enabled == null || !enabled.equalsIgnoreCase("true")) { + logger.info("disable iceberg test") + return + } + + String restPort = context.config.otherConfigs.get("iceberg_rest_uri_port") + String minioPort = context.config.otherConfigs.get("iceberg_minio_port") + String externalEnvIp = context.config.otherConfigs.get("externalEnvIp") + String catalogName = "test_iceberg_schema_dual_relation_matrix" + String dbName = "iceberg_schema_dual_relation_db" + String tableName = "dual_schema_timeline" + + def latestSnapshotId = { + List> rows = spark_iceberg """ + select snapshot_id + from demo.${dbName}.${tableName}.snapshots + order by committed_at desc + limit 1 + """ + assertEquals(1, rows.size()) + return rows[0][0].toString() + } + + sql """drop catalog if exists ${catalogName}""" + sql """ + create catalog ${catalogName} properties ( + 'type'='iceberg', + 'iceberg.catalog.type'='rest', + 'uri'='http://${externalEnvIp}:${restPort}', + 's3.access_key'='admin', + 's3.secret_key'='password', + 's3.endpoint'='http://${externalEnvIp}:${minioPort}', + 's3.region'='us-east-1' + ) + """ + + try { + spark_iceberg_multi """ + create database if not exists demo.${dbName}; + drop table if exists demo.${dbName}.${tableName}; + create table demo.${dbName}.${tableName} ( + id int, + old_name string, + info struct + ) using iceberg + tblproperties ('format-version'='2', 'write.format.default'='parquet'); + insert into demo.${dbName}.${tableName} + values (1, 'old-1', named_struct('added', 10, 'keep', 11)); + """ + String oldSnapshot = latestSnapshotId() + + spark_iceberg_multi """ + alter table demo.${dbName}.${tableName} rename column old_name to new_name; + alter table demo.${dbName}.${tableName} + rename column info.added to renamed; + insert into demo.${dbName}.${tableName} + values (2, 'new-2', named_struct('renamed', 20, 'keep', 21)); + """ + String newSnapshot = latestSnapshotId() + + sql """switch ${catalogName}""" + sql """use ${dbName}""" + + // Scenario TC07-baseline: each historical relation resolves its own schema in isolation. + assertEquals([[1, "old-1", 10]], sql(""" + select id, old_name, info.added + from ${tableName} for version as of ${oldSnapshot} + order by id + """)) + assertEquals([[1, "old-1", 10], [2, "new-2", 20]], sql(""" + select id, new_name, info.renamed + from ${tableName} for version as of ${newSnapshot} + order by id + """)) + + // Scenario TC07-join negative contract: + // two historical relations in one statement currently reuse the first schema. + test { + sql """ + select o.id, o.old_name, n.new_name + from ( + select id, old_name + from ${tableName} for version as of ${oldSnapshot} + ) o + join ( + select id, new_name + from ${tableName} for version as of ${newSnapshot} + ) n on o.id = n.id + order by o.id + """ + exception "Unknown column 'new_name'" + } + + // Scenario TC07-reverse-join: binding must be independent of relation order. + test { + sql """ + select n.id, n.new_name, o.old_name + from ( + select id, new_name + from ${tableName} for version as of ${newSnapshot} + ) n + join ( + select id, old_name + from ${tableName} for version as of ${oldSnapshot} + ) o on n.id = o.id + order by n.id + """ + exception "Unknown column 'old_name'" + } + + // Scenario TC07-union: top-level historical schemas stay relation-local. + test { + sql """ + select id, old_name as name_value + from ${tableName} for version as of ${oldSnapshot} + union all + select id, new_name as name_value + from ${tableName} for version as of ${newSnapshot} + order by id, name_value + """ + exception "Unknown column 'new_name'" + } + + // Scenario TC07-nested-union: nested field lookup is also relation-local. + test { + sql """ + select id, info.added as nested_value + from ${tableName} for version as of ${oldSnapshot} + union all + select id, info.renamed as nested_value + from ${tableName} for version as of ${newSnapshot} + order by id, nested_value + """ + exception "No such struct field 'renamed'" + } + + // Scenario TC07-CTE: CTE boundaries must not collapse snapshot schemas. + test { + sql """ + with old_ref as ( + select id, old_name + from ${tableName} for version as of ${oldSnapshot} + ), new_ref as ( + select id, new_name + from ${tableName} for version as of ${newSnapshot} + ) + select old_ref.id, old_ref.old_name, new_ref.new_name + from old_ref join new_ref on old_ref.id = new_ref.id + order by old_ref.id + """ + exception "Unknown column 'new_name'" + } + + // Scenario TC07-correlated-subquery: subqueries require an independent schema. + test { + sql """ + select o.id, o.old_name + from ${tableName} for version as of ${oldSnapshot} o + where exists ( + select 1 + from ${tableName} for version as of ${newSnapshot} n + where n.id = o.id and n.new_name is not null + ) + order by o.id + """ + exception "Unknown column 'new_name'" + } + } finally { + sql """drop catalog if exists ${catalogName}""" + } +} diff --git a/regression-test/suites/external_table_p0/iceberg/test_iceberg_schema_equality_delete_time_travel.groovy b/regression-test/suites/external_table_p0/iceberg/test_iceberg_schema_equality_delete_time_travel.groovy new file mode 100644 index 00000000000000..5e9ab170b5bb6f --- /dev/null +++ b/regression-test/suites/external_table_p0/iceberg/test_iceberg_schema_equality_delete_time_travel.groovy @@ -0,0 +1,145 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +suite("test_iceberg_schema_equality_delete_time_travel", + "p0,external,iceberg,external_docker,external_docker_iceberg") { + String enabled = context.config.otherConfigs.get("enableIcebergTest") + if (enabled == null || !enabled.equalsIgnoreCase("true")) { + logger.info("disable iceberg test") + return + } + + String restPort = context.config.otherConfigs.get("iceberg_rest_uri_port") + String minioPort = context.config.otherConfigs.get("iceberg_minio_port") + String externalEnvIp = context.config.otherConfigs.get("externalEnvIp") + String catalogName = "test_iceberg_schema_equality_delete_time_travel" + + sql """drop catalog if exists ${catalogName}""" + sql """ + create catalog ${catalogName} properties ( + 'type'='iceberg', + 'iceberg.catalog.type'='rest', + 'uri'='http://${externalEnvIp}:${restPort}', + 's3.access_key'='admin', + 's3.secret_key'='password', + 's3.endpoint'='http://${externalEnvIp}:${minioPort}', + 's3.region'='us-east-1' + ) + """ + sql """switch ${catalogName}""" + sql """use multi_catalog""" + + try { + ["par", "orc"].each { format -> + String tableName = "equality_delete_${format}_1" + List snapshotIds = sql(""" + select snapshot_id + from ${tableName}\$snapshots + order by committed_at, snapshot_id + """).collect { row -> row[0].toString() } + assertTrue(snapshotIds.size() >= 7, + "Equality-delete fixture must retain the schema timeline for ${tableName}") + + String oldSnapshot = snapshotIds[0] + String renamedSnapshot = snapshotIds[2] + String latestSnapshot = snapshotIds.last() + + // Scenario TC04-EQ-S04: equality deletes before rename stay bound to the old field ID. + assertEquals([ + [1, "smith", "a"], + [2, "danny", "b"], + [3, "alice", "c"], + [4, "bob", "d"] + ], sql(""" + select id, name, data + from ${tableName} for version as of ${oldSnapshot} + order by id + """)) + test { + sql """ + select new_new_id + from ${tableName} for version as of ${oldSnapshot} + """ + exception "Unknown column 'new_new_id'" + } + + // Scenario TC04-EQ-S04/S08: the renamed-key snapshot applies equality deletes by field ID. + List> renamedRows = sql(""" + select new_id, name, data + from ${tableName} for version as of ${renamedSnapshot} + order by new_id, name, data + """) + String lastName = format == "par" ? "parker" : "orcker" + assertEquals([ + [1, "smith2", "aa"], + [2, "danny2", "bb"], + [3, "alice", "c"], + [4, "bob", "e"], + [5, "dennis", "f"], + [6, "jasson", "g"], + [7, lastName, "h"] + ], renamedRows) + + // Scenario TC04-EQ-S07: re-added id has a new field ID and must not match old delete keys. + List> latestRowsV2 + sql """set enable_file_scanner_v2=true""" + latestRowsV2 = sql(""" + select new_new_id, new_name, data, id + from ${tableName} for version as of ${latestSnapshot} + order by new_new_id, new_name, data, id + """) + assertEquals([ + [1, "smith4", "aaaa", 1], + [2, "danny2", "bb", null], + [3, "alice", "c", null], + [4, "bob3", "eee", null], + [5, "dennis2", "ff", null], + [6, "jasson", "g", null], + [7, lastName, "h", null] + ], latestRowsV2) + + // Scenario TC04-EQ-R13: legacy and V2 scanners apply the same equality deletes. + sql """set enable_file_scanner_v2=false""" + List> latestRowsLegacy = sql(""" + select new_new_id, new_name, data, id + from ${tableName} for version as of ${latestSnapshot} + order by new_new_id, new_name, data, id + """) + assertEquals(latestRowsV2, latestRowsLegacy) + + // Scenario TC04-EQ-query-shapes: projection, predicate and aggregation share delete semantics. + assertEquals(latestRowsLegacy.size().toLong(), sql(""" + select count(*) from ${tableName} for version as of ${latestSnapshot} + """)[0][0]) + assertEquals(latestRowsLegacy.findAll { row -> row[0].toString().toInteger() >= 4 }.size().toLong(), + sql(""" + select count(*) + from ${tableName} for version as of ${latestSnapshot} + where new_new_id >= 4 + """)[0][0]) + + // Old refs remain readable after all equality-delete/schema commits. + assertEquals(4L, sql(""" + select count(*) + from ${tableName} for version as of ${oldSnapshot} + """)[0][0]) + } + } finally { + sql """set enable_file_scanner_v2=true""" + sql """drop catalog if exists ${catalogName}""" + } +} diff --git a/regression-test/suites/external_table_p0/iceberg/test_iceberg_schema_evolution_filter_binding.groovy b/regression-test/suites/external_table_p0/iceberg/test_iceberg_schema_evolution_filter_binding.groovy new file mode 100644 index 00000000000000..1500b6702ae1b3 --- /dev/null +++ b/regression-test/suites/external_table_p0/iceberg/test_iceberg_schema_evolution_filter_binding.groovy @@ -0,0 +1,119 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +suite("test_iceberg_schema_evolution_filter_binding", "p0,external") { + String enabled = context.config.otherConfigs.get("enableIcebergTest") + if (enabled == null || !enabled.equalsIgnoreCase("true")) { + logger.info("disable iceberg test.") + return + } + + String catalogName = "test_iceberg_schema_evolution_filter_binding" + String dbName = "schema_evolution_filter_binding_db" + String tableName = "schema_evolution_filter_binding" + String restPort = context.config.otherConfigs.get("iceberg_rest_uri_port") + String minioPort = context.config.otherConfigs.get("iceberg_minio_port") + String externalEnvIp = context.config.otherConfigs.get("externalEnvIp") + + sql """drop catalog if exists ${catalogName}""" + sql """ + create catalog ${catalogName} properties ( + 'type' = 'iceberg', + 'iceberg.catalog.type' = 'rest', + 'uri' = 'http://${externalEnvIp}:${restPort}', + 's3.access_key' = 'admin', + 's3.secret_key' = 'password', + 's3.endpoint' = 'http://${externalEnvIp}:${minioPort}', + 's3.region' = 'us-east-1' + ) + """ + + try { + sql """switch ${catalogName}""" + sql """drop database if exists ${dbName} force""" + sql """create database ${dbName}""" + sql """use ${dbName}""" + sql """set enable_fallback_to_original_planner = false""" + sql """set runtime_filter_mode = 'GLOBAL'""" + + sql """ + create table ${tableName} ( + id int, + old_name string, + dropped_value string, + metric int + ) + """ + sql """ + insert into ${tableName} values + (1, 'old-a', 'drop-a', 10), + (2, 'old-b', 'drop-b', 20), + (3, 'old-c', 'drop-c', 30) + """ + + // Warm the scan metadata before changing field names and types. Predicates must continue + // to bind by Iceberg field ID rather than by a cached ordinal from the old schema. + order_qt_schema_filter_warm """ + select id, old_name, metric from ${tableName} order by id + """ + sql """alter table ${tableName} rename column old_name new_name""" + sql """alter table ${tableName} drop column dropped_value""" + sql """alter table ${tableName} modify column metric bigint""" + sql """ + insert into ${tableName} values + (4, 'new-d', 4000000000), + (5, 'new-e', 5000000000) + """ + + order_qt_schema_filter_static """ + select id, new_name, metric + from ${tableName} + where new_name in ('old-b', 'new-d') and metric >= 20 + order by id + """ + order_qt_schema_filter_runtime """ + with filter_keys as ( + select 2 as id + union all + select 4 + ) + select t.id, t.new_name, t.metric + from ${tableName} t + join filter_keys k on t.id = k.id + order by t.id + """ + qt_schema_filter_aggregate """ + with filter_keys as ( + select 'old-b' as new_name + union all + select 'new-e' + ) + select count(*), sum(t.metric) + from ${tableName} t + join filter_keys k on t.new_name = k.new_name + """ + order_qt_schema_filter_null_semantics """ + select id, new_name + from ${tableName} + where new_name is not null and metric between 20 and 4000000000 + order by id + """ + } finally { + sql """drop database if exists ${catalogName}.${dbName} force""" + sql """drop catalog if exists ${catalogName}""" + } +} diff --git a/regression-test/suites/external_table_p0/iceberg/test_iceberg_schema_metadata_atomicity_matrix.groovy b/regression-test/suites/external_table_p0/iceberg/test_iceberg_schema_metadata_atomicity_matrix.groovy new file mode 100644 index 00000000000000..45307819e599a2 --- /dev/null +++ b/regression-test/suites/external_table_p0/iceberg/test_iceberg_schema_metadata_atomicity_matrix.groovy @@ -0,0 +1,163 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +suite("test_iceberg_schema_metadata_atomicity_matrix", + "p0,external,iceberg,external_docker,external_docker_iceberg") { + String enabled = context.config.otherConfigs.get("enableIcebergTest") + if (enabled == null || !enabled.equalsIgnoreCase("true")) { + logger.info("disable iceberg test") + return + } + + String restPort = context.config.otherConfigs.get("iceberg_rest_uri_port") + String minioPort = context.config.otherConfigs.get("iceberg_minio_port") + String externalEnvIp = context.config.otherConfigs.get("externalEnvIp") + String catalogName = "test_iceberg_schema_metadata_atomicity_matrix" + String dbName = "iceberg_schema_metadata_atomicity_db" + String tableName = "metadata_timeline" + + def snapshotCount = { + return sql("""select count(*) from ${tableName}\$snapshots""")[0][0].toString().toInteger() + } + def schemaCount = { + return sql("""select count(*) from ${tableName}\$metadata_log_entries""")[0][0] + .toString().toInteger() + } + def allDescText = { + return sql("""desc ${tableName}""").flatten().collect { + it == null ? "" : it.toString() + }.join(" ") + } + + sql """drop catalog if exists ${catalogName}""" + sql """ + create catalog ${catalogName} properties ( + 'type'='iceberg', + 'iceberg.catalog.type'='rest', + 'uri'='http://${externalEnvIp}:${restPort}', + 's3.access_key'='admin', + 's3.secret_key'='password', + 's3.endpoint'='http://${externalEnvIp}:${minioPort}', + 's3.region'='us-east-1' + ) + """ + sql """switch ${catalogName}""" + sql """create database if not exists ${dbName}""" + sql """use ${dbName}""" + + try { + sql """drop table if exists ${tableName}""" + sql """ + create table ${tableName} ( + id int not null, + required_name string not null, + optional_value int, + info struct, + attrs map + ) engine=iceberg + properties ('format-version'='2', 'write.format.default'='parquet') + """ + sql """ + insert into ${tableName} values + (1, 'name-1', 10, named_struct('value', 100, 'keep', 101), map('k', 1000)) + """ + sql """alter table ${tableName} create tag metadata_before_change""" + int initialSnapshots = snapshotCount() + + // Scenario S19-comment: top-level and nested comments are metadata-only schema changes. + sql """alter table ${tableName} modify column required_name comment 'top-level-comment'""" + sql """alter table ${tableName} modify column info.value comment 'nested-comment'""" + String descAfterComment = allDescText() + List> sparkDescription = spark_iceberg(""" + describe table extended demo.${dbName}.${tableName} + """) + String sparkDescriptionText = sparkDescription.flatten().collect { + it == null ? "" : it.toString() + }.join(" ") + assertTrue(sparkDescriptionText.contains("top-level-comment")) + // Negative contract: Doris DESC currently omits Iceberg field comments. + assertFalse(descAfterComment.contains("top-level-comment")) + assertFalse(descAfterComment.contains("nested-comment")) + assertEquals(initialSnapshots, snapshotCount()) + + // Scenario S19-nullability: relaxing required to optional preserves data and historical refs. + sql """alter table ${tableName} modify column required_name string null""" + assertEquals([[1, "name-1", 100]], sql(""" + select id, required_name, info.value from ${tableName} order by id + """)) + assertEquals([[1, "name-1", 100]], sql(""" + select id, required_name, info.value + from ${tableName}@tag(metadata_before_change) + order by id + """)) + assertEquals(initialSnapshots, snapshotCount()) + + // Scenario S20-required-add: Iceberg rejects non-null additions and commits no metadata. + int metadataBeforeRequiredAdd = schemaCount() + test { + sql """alter table ${tableName} add column required_added int not null""" + exception "default value" + } + assertEquals(metadataBeforeRequiredAdd, schemaCount()) + + // Scenario S20-default: v2 non-null initial defaults are unsupported and atomic. + int metadataBeforeDefault = schemaCount() + test { + sql """ + alter table ${tableName} + add column default_added string default 'default-value' + """ + exception "non-null default" + } + assertEquals(metadataBeforeDefault, schemaCount()) + + // Scenario S20-nullability-strengthen: optional fields cannot become required with old NULLs. + int metadataBeforeNotNull = schemaCount() + test { + sql """alter table ${tableName} modify column optional_value int not null""" + exception "not null" + } + assertEquals(metadataBeforeNotNull, schemaCount()) + + // Scenario S20-narrowing: illegal type narrowing is rejected without a schema commit. + int metadataBeforeNarrowing = schemaCount() + test { + sql """alter table ${tableName} modify column optional_value smallint""" + exception "not supported for Iceberg column" + } + assertEquals(metadataBeforeNarrowing, schemaCount()) + + // Scenario S20-map-key: map keys cannot be evolved independently. + int metadataBeforeMapKey = schemaCount() + test { + sql """alter table ${tableName} modify column attrs.`key` bigint""" + exception "Cannot modify MAP key nested column" + } + assertEquals(metadataBeforeMapKey, schemaCount()) + + sql """refresh table ${tableName}""" + assertEquals([[1, "name-1", 10, 100, 1000]], sql(""" + select id, required_name, optional_value, info.value, attrs['k'] + from ${tableName} + order by id + """)) + assertEquals(initialSnapshots, snapshotCount()) + } finally { + sql """drop database if exists ${dbName} force""" + sql """drop catalog if exists ${catalogName}""" + } +} diff --git a/regression-test/suites/external_table_p0/iceberg/test_iceberg_schema_position_dv_time_travel.groovy b/regression-test/suites/external_table_p0/iceberg/test_iceberg_schema_position_dv_time_travel.groovy new file mode 100644 index 00000000000000..1afc70bc5f2074 --- /dev/null +++ b/regression-test/suites/external_table_p0/iceberg/test_iceberg_schema_position_dv_time_travel.groovy @@ -0,0 +1,180 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +suite("test_iceberg_schema_position_dv_time_travel", + "p0,external,iceberg,external_docker,external_docker_iceberg") { + String enabled = context.config.otherConfigs.get("enableIcebergTest") + if (enabled == null || !enabled.equalsIgnoreCase("true")) { + logger.info("disable iceberg test") + return + } + + String restPort = context.config.otherConfigs.get("iceberg_rest_uri_port") + String minioPort = context.config.otherConfigs.get("iceberg_minio_port") + String externalEnvIp = context.config.otherConfigs.get("externalEnvIp") + String catalogName = "test_iceberg_schema_position_dv_time_travel" + String dbName = "iceberg_schema_position_dv_db" + + def latestSnapshotId = { String tableName -> + return spark_iceberg(""" + select snapshot_id + from demo.${dbName}.${tableName}.snapshots + order by committed_at desc + limit 1 + """)[0][0].toString() + } + + sql """drop catalog if exists ${catalogName}""" + sql """ + create catalog ${catalogName} properties ( + 'type'='iceberg', + 'iceberg.catalog.type'='rest', + 'uri'='http://${externalEnvIp}:${restPort}', + 's3.access_key'='admin', + 's3.secret_key'='password', + 's3.endpoint'='http://${externalEnvIp}:${minioPort}', + 's3.region'='us-east-1' + ) + """ + + try { + spark_iceberg """create database if not exists demo.${dbName}""" + + [ + [table: "position_parquet", version: "2", format: "parquet"], + [table: "position_orc", version: "2", format: "orc"], + [table: "dv_parquet", version: "3", format: "parquet"], + [table: "dv_orc", version: "3", format: "orc"] + ].each { profile -> + String tableName = profile.table + spark_iceberg_multi """ + drop table if exists demo.${dbName}.${tableName}; + create table demo.${dbName}.${tableName} ( + id int, + old_name string, + victim string, + metric int, + info struct + ) using iceberg + tblproperties ( + 'format-version'='${profile.version}', + 'write.format.default'='${profile.format}', + 'write.delete.mode'='merge-on-read', + 'write.update.mode'='merge-on-read', + 'write.merge.mode'='merge-on-read', + 'write.distribution-mode'='none', + 'write.target-file-size-bytes'='134217728' + ); + insert into demo.${dbName}.${tableName} + select /*+ COALESCE(1) */ id, old_name, victim, metric, info from values + (1, 'old-1', 'victim-1', 10, + named_struct('old_child', 100, 'keep', 101)), + (2, 'old-2', 'victim-2', 20, + named_struct('old_child', 200, 'keep', 201)), + (3, 'old-3', 'victim-3', 30, + named_struct('old_child', 300, 'keep', 301)) + as t(id, old_name, victim, metric, info); + """ + String beforeDelete = latestSnapshotId(tableName) + + // Scenario TC04-position/DV-before-change: delete visibility is snapshot-local. + spark_iceberg """ + delete from demo.${dbName}.${tableName} where id = 2 + """ + String afterFirstDelete = latestSnapshotId(tableName) + + spark_iceberg_multi """ + alter table demo.${dbName}.${tableName} rename column old_name to new_name; + alter table demo.${dbName}.${tableName} + rename column info.old_child to new_child; + alter table demo.${dbName}.${tableName} drop column victim; + alter table demo.${dbName}.${tableName} add column victim bigint; + alter table demo.${dbName}.${tableName} alter column metric type bigint; + insert into demo.${dbName}.${tableName} + (id, new_name, victim, metric, info) values + (4, 'new-4', 4000, 6000000000, + named_struct('new_child', 400, 'keep', 401)); + delete from demo.${dbName}.${tableName} where id = 3; + """ + String afterSchemaDelete = latestSnapshotId(tableName) + + sql """switch ${catalogName}""" + sql """use ${dbName}""" + sql """refresh table ${tableName}""" + + assertEquals([ + [1, "old-1", "victim-1", 10, 100], + [2, "old-2", "victim-2", 20, 200], + [3, "old-3", "victim-3", 30, 300] + ], sql(""" + select id, old_name, victim, metric, info.old_child + from ${tableName} for version as of ${beforeDelete} + order by id + """)) + assertEquals([[1], [3]], sql(""" + select id + from ${tableName} for version as of ${afterFirstDelete} + order by id + """)) + + // Scenario TC04-S04/S06/S07/S08/S10: delete files survive top-level and nested evolution. + assertEquals([ + [1, "old-1", null, 10L, 100], + [4, "new-4", 4000L, 6000000000L, 400] + ], sql(""" + select id, new_name, victim, metric, info.new_child + from ${tableName} for version as of ${afterSchemaDelete} + order by id + """)) + + // Scenario TC04-field-ID: re-added victim never exposes values from the dropped STRING field. + assertEquals([[1, null], [4, 4000L]], sql(""" + select id, victim + from ${tableName} for version as of ${afterSchemaDelete} + order by id + """)) + + // Scenario TC04-reader-diff: both scanners apply position deletes or DVs identically. + sql """set enable_file_scanner_v2=true""" + List> v2Rows = sql(""" + select id, new_name, victim, metric, info.new_child + from ${tableName} for version as of ${afterSchemaDelete} + where metric >= 10 + order by id + """) + sql """set enable_file_scanner_v2=false""" + List> legacyRows = sql(""" + select id, new_name, victim, metric, info.new_child + from ${tableName} for version as of ${afterSchemaDelete} + where metric >= 10 + order by id + """) + assertEquals(v2Rows, legacyRows) + + // Scenario TC04-delete-file-kind: v2 produces position deletes and v3 produces DVs. + List> deleteFiles = spark_iceberg(""" + select content, file_format + from demo.${dbName}.${tableName}.delete_files + """) + assertTrue(deleteFiles.size() > 0, + "The ${profile.version == '3' ? 'DV' : 'position-delete'} profile needs delete files") + } + } finally { + sql """set enable_file_scanner_v2=true""" + sql """drop catalog if exists ${catalogName}""" + } +} diff --git a/regression-test/suites/external_table_p0/iceberg/test_iceberg_schema_ref_actions_matrix.groovy b/regression-test/suites/external_table_p0/iceberg/test_iceberg_schema_ref_actions_matrix.groovy new file mode 100644 index 00000000000000..607895a6e21fb4 --- /dev/null +++ b/regression-test/suites/external_table_p0/iceberg/test_iceberg_schema_ref_actions_matrix.groovy @@ -0,0 +1,242 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +suite("test_iceberg_schema_ref_actions_matrix", + "p0,external,iceberg,external_docker,external_docker_iceberg") { + String enabled = context.config.otherConfigs.get("enableIcebergTest") + if (enabled == null || !enabled.equalsIgnoreCase("true")) { + logger.info("disable iceberg test") + return + } + + String restPort = context.config.otherConfigs.get("iceberg_rest_uri_port") + String minioPort = context.config.otherConfigs.get("iceberg_minio_port") + String externalEnvIp = context.config.otherConfigs.get("externalEnvIp") + String catalogName = "test_iceberg_schema_ref_actions_matrix" + String dbName = "iceberg_schema_ref_actions_db" + + def snapshots = { String tableName -> + return sql(""" + select snapshot_id + from ${tableName}\$snapshots + order by committed_at, snapshot_id + """).collect { row -> row[0].toString() } + } + + def assertUnknownColumn = { String query, String columnName -> + test { + sql query + exception "'${columnName}'" + } + } + + sql """drop catalog if exists ${catalogName}""" + sql """ + create catalog ${catalogName} properties ( + 'type'='iceberg', + 'iceberg.catalog.type'='rest', + 'uri'='http://${externalEnvIp}:${restPort}', + 's3.access_key'='admin', + 's3.secret_key'='password', + 's3.endpoint'='http://${externalEnvIp}:${minioPort}', + 's3.region'='us-east-1' + ) + """ + sql """switch ${catalogName}""" + sql """create database if not exists ${dbName}""" + sql """use ${dbName}""" + + try { + // Scenario T15: rollback_to_snapshot rolls data back while the current schema stays latest. + String rollbackTable = "rollback_schema_timeline" + sql """drop table if exists ${rollbackTable}""" + sql """ + create table ${rollbackTable} ( + id int, + old_name string, + payload struct + ) engine=iceberg + properties ('format-version'='2', 'write.format.default'='parquet') + """ + sql """ + insert into ${rollbackTable} + values (1, 'old-1', named_struct('old_child', 10, 'keep', 11)) + """ + String rollbackOldSnapshot = snapshots(rollbackTable).last() + sql """alter table ${rollbackTable} create tag rollback_old""" + + sql """alter table ${rollbackTable} rename column old_name new_name""" + sql """alter table ${rollbackTable} rename column payload.old_child new_child""" + sql """ + insert into ${rollbackTable} + values (2, 'new-2', named_struct('new_child', 20, 'keep', 21)) + """ + String rollbackNewSnapshot = snapshots(rollbackTable).last() + sql """alter table ${rollbackTable} create tag rollback_new""" + + assertEquals([[1, "old-1", 10]], sql(""" + select id, old_name, payload.old_child + from ${rollbackTable} for version as of ${rollbackOldSnapshot} + order by id + """)) + assertEquals([[1, "old-1", 10], [2, "new-2", 20]], sql(""" + select id, new_name, payload.new_child + from ${rollbackTable} for version as of ${rollbackNewSnapshot} + order by id + """)) + + sql """ + alter table ${rollbackTable} + execute rollback_to_snapshot('snapshot_id'='${rollbackOldSnapshot}') + """ + sql """refresh table ${rollbackTable}""" + assertEquals([[1, "old-1", 10]], sql(""" + select id, new_name, payload.new_child + from ${rollbackTable} + order by id + """)) + assertUnknownColumn("""select old_name from ${rollbackTable}""", "old_name") + assertEquals([[1, "old-1", 10]], sql(""" + select id, old_name, payload.old_child + from ${rollbackTable}@tag(rollback_old) + order by id + """)) + assertEquals([[1, "old-1", 10], [2, "new-2", 20]], sql(""" + select id, new_name, payload.new_child + from ${rollbackTable}@tag(rollback_new) + order by id + """)) + + // Scenario T16-cherrypick: append snapshot with the renamed schema can be replayed after rollback. + sql """ + alter table ${rollbackTable} + execute cherrypick_snapshot('snapshot_id'='${rollbackNewSnapshot}') + """ + sql """refresh table ${rollbackTable}""" + assertEquals([[1, "old-1", 10], [2, "new-2", 20]], sql(""" + select id, new_name, payload.new_child + from ${rollbackTable} + order by id + """)) + assertEquals([[1, "old-1", 10]], sql(""" + select id, old_name, payload.old_child + from ${rollbackTable}@tag(rollback_old) + order by id + """)) + + // Scenario T15-timestamp: timestamp rollback also keeps the latest current schema. + String timestampTable = "rollback_timestamp_schema_timeline" + sql """drop table if exists ${timestampTable}""" + sql """ + create table ${timestampTable} ( + id int, + old_name string + ) engine=iceberg + properties ('format-version'='2', 'write.format.default'='orc') + """ + sql """insert into ${timestampTable} values (1, 'old-1')""" + List> timestampCheckpoint = sql(""" + select snapshot_id, + date_format(date_add(committed_at, interval 1 second), + '%Y-%m-%d %H:%i:%s.000') + from ${timestampTable}\$snapshots + order by committed_at desc + limit 1 + """) + String timestampOldSnapshot = timestampCheckpoint[0][0].toString() + String rollbackTimestamp = timestampCheckpoint[0][1].toString() + sql """alter table ${timestampTable} create tag timestamp_old""" + Thread.sleep(1100) + sql """alter table ${timestampTable} rename column old_name new_name""" + sql """insert into ${timestampTable} values (2, 'new-2')""" + + sql """ + alter table ${timestampTable} + execute rollback_to_timestamp('timestamp'='${rollbackTimestamp}') + """ + sql """refresh table ${timestampTable}""" + assertEquals([[1, "old-1"]], sql(""" + select id, new_name from ${timestampTable} order by id + """)) + assertEquals([[1, "old-1"]], sql(""" + select id, old_name + from ${timestampTable} for version as of ${timestampOldSnapshot} + order by id + """)) + + // Scenario T16-fast-forward: advance a pre-rename branch to the renamed main schema. + String fastForwardTable = "fast_forward_schema_timeline" + sql """drop table if exists ${fastForwardTable}""" + sql """ + create table ${fastForwardTable} ( + id int, + old_name string, + metric int + ) engine=iceberg + properties ('format-version'='2', 'write.format.default'='parquet') + """ + sql """insert into ${fastForwardTable} values (1, 'old-1', 10)""" + sql """alter table ${fastForwardTable} create branch pre_rename_branch""" + sql """alter table ${fastForwardTable} create tag pre_rename_tag""" + sql """alter table ${fastForwardTable} rename column old_name new_name""" + sql """alter table ${fastForwardTable} modify column metric bigint""" + sql """insert into ${fastForwardTable} values (2, 'new-2', 6000000000)""" + + // Scenario T08 negative contract: before fast-forward, branch reads use the latest rename schema. + test { + sql """ + select id, old_name, metric + from ${fastForwardTable}@branch(pre_rename_branch) + order by id + """ + exception "Unknown column 'old_name'" + } + assertEquals([[1, "old-1", 10]], sql(""" + select id, old_name, metric + from ${fastForwardTable}@tag(pre_rename_tag) + order by id + """)) + + // Scenario T09 negative contract: a pre-rename branch write uses main's latest schema. + test { + sql """ + insert into ${fastForwardTable}@branch(pre_rename_branch) + (id, old_name, metric) values (3, 'branch-3', 30) + """ + exception "Unknown column 'old_name'" + } + + sql """ + alter table ${fastForwardTable} + execute fast_forward('branch'='pre_rename_branch', 'to'='main') + """ + sql """refresh table ${fastForwardTable}""" + assertEquals([[1, "old-1", 10L], [2, "new-2", 6000000000L]], sql(""" + select id, new_name, metric + from ${fastForwardTable}@branch(pre_rename_branch) + order by id + """)) + assertEquals([[1, "old-1", 10]], sql(""" + select id, old_name, metric + from ${fastForwardTable}@tag(pre_rename_tag) + order by id + """)) + } finally { + sql """drop database if exists ${dbName} force""" + sql """drop catalog if exists ${catalogName}""" + } +} diff --git a/regression-test/suites/external_table_p0/iceberg/test_iceberg_schema_time_travel_matrix.groovy b/regression-test/suites/external_table_p0/iceberg/test_iceberg_schema_time_travel_matrix.groovy new file mode 100644 index 00000000000000..e0876e17ea5833 --- /dev/null +++ b/regression-test/suites/external_table_p0/iceberg/test_iceberg_schema_time_travel_matrix.groovy @@ -0,0 +1,732 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +suite("test_iceberg_schema_time_travel_matrix", + "p0,external,iceberg,external_docker,external_docker_iceberg") { + String enabled = context.config.otherConfigs.get("enableIcebergTest") + if (enabled == null || !enabled.equalsIgnoreCase("true")) { + logger.info("disable iceberg test") + return + } + + String restPort = context.config.otherConfigs.get("iceberg_rest_uri_port") + String minioPort = context.config.otherConfigs.get("iceberg_minio_port") + String externalEnvIp = context.config.otherConfigs.get("externalEnvIp") + String catalogName = "test_iceberg_schema_time_travel_matrix" + String noCacheCatalogName = "test_iceberg_schema_time_travel_matrix_no_cache" + String dbName = "iceberg_schema_time_travel_matrix_db" + String topTable = "top_timeline" + String nestedTable = "nested_timeline" + String dorisNestedTable = "doris_nested_timeline" + String deleteTable = "delete_partition_timeline" + + def latestSnapshotId = { String tableName -> + List> rows = spark_iceberg """ + SELECT snapshot_id + FROM demo.${dbName}.${tableName}.snapshots + ORDER BY committed_at DESC + LIMIT 1 + """ + assertEquals(1, rows.size()) + return rows[0][0].toString() + } + + def assertUnknownColumn = { String query, String columnName -> + test { + sql query + exception "'${columnName}'" + } + } + + sql """drop catalog if exists ${catalogName}""" + sql """drop catalog if exists ${noCacheCatalogName}""" + sql """ + CREATE CATALOG ${catalogName} PROPERTIES ( + 'type'='iceberg', + 'iceberg.catalog.type'='rest', + 'uri'='http://${externalEnvIp}:${restPort}', + 's3.access_key'='admin', + 's3.secret_key'='password', + 's3.endpoint'='http://${externalEnvIp}:${minioPort}', + 's3.region'='us-east-1' + ) + """ + sql """ + CREATE CATALOG ${noCacheCatalogName} PROPERTIES ( + 'type'='iceberg', + 'iceberg.catalog.type'='rest', + 'uri'='http://${externalEnvIp}:${restPort}', + 's3.access_key'='admin', + 's3.secret_key'='password', + 's3.endpoint'='http://${externalEnvIp}:${minioPort}', + 's3.region'='us-east-1', + 'meta.cache.iceberg.table.ttl-second'='0', + 'meta.cache.iceberg.schema.ttl-second'='0' + ) + """ + + try { + spark_iceberg_multi """ + CREATE DATABASE IF NOT EXISTS demo.${dbName}; + DROP TABLE IF EXISTS demo.${dbName}.${topTable}; + CREATE TABLE demo.${dbName}.${topTable} ( + id INT, + old_name STRING, + victim STRING, + metric INT + ) USING iceberg + TBLPROPERTIES ( + 'format-version'='2', + 'write.format.default'='parquet' + ); + INSERT INTO demo.${dbName}.${topTable} + VALUES (1, 'alpha', 'old-v1', 10); + """ + String topCp0 = latestSnapshotId(topTable) + sql """ALTER TABLE `${catalogName}`.`${dbName}`.`${topTable}` CREATE TAG top_cp0""" + sql """ALTER TABLE `${catalogName}`.`${dbName}`.`${topTable}` CREATE BRANCH top_cp0_branch""" + Thread.sleep(1100) + + // Scenario S01/S02/S03 x T00/T01/T02/T05/T08: + // add and reorder columns while keeping the pre-change snapshot, tag and branch readable. + spark_iceberg_multi """ + ALTER TABLE demo.${dbName}.${topTable} + ADD COLUMNS (added STRING AFTER old_name, added_second BIGINT); + ALTER TABLE demo.${dbName}.${topTable} ALTER COLUMN added_second FIRST; + INSERT INTO demo.${dbName}.${topTable} + (id, old_name, victim, metric, added, added_second) + VALUES (2, 'beta', 'old-v2', 20, 'added-v2', 200); + """ + String topCpAdd = latestSnapshotId(topTable) + sql """ALTER TABLE `${catalogName}`.`${dbName}`.`${topTable}` CREATE TAG top_cp_add""" + Thread.sleep(1100) + + // Scenario S04/S05 x T00-T08: + // a rename must be observable through explicit old/new names, not only SELECT *. + spark_iceberg_multi """ + ALTER TABLE demo.${dbName}.${topTable} RENAME COLUMN old_name TO MixedName; + INSERT INTO demo.${dbName}.${topTable} + (id, MixedName, victim, metric, added, added_second) + VALUES (3, 'gamma', 'old-v3', 30, 'added-v3', 300); + """ + String topCpRename = latestSnapshotId(topTable) + sql """ALTER TABLE `${catalogName}`.`${dbName}`.`${topTable}` CREATE TAG top_cp_rename""" + Thread.sleep(1100) + + // Scenario S06 x T00/T01/T02/T05: old refs retain the dropped field. + spark_iceberg_multi """ + ALTER TABLE demo.${dbName}.${topTable} DROP COLUMN victim; + INSERT INTO demo.${dbName}.${topTable} + (id, MixedName, metric, added, added_second) + VALUES (4, 'delta', 40, 'added-v4', 400); + """ + String topCpDrop = latestSnapshotId(topTable) + sql """ALTER TABLE `${catalogName}`.`${dbName}`.`${topTable}` CREATE TAG top_cp_drop""" + Thread.sleep(1100) + + // Scenario S07 x T00/T01/T02/T05/T12/T13: + // reusing the name with a new BIGINT field ID must never expose old STRING values. + spark_iceberg_multi """ + ALTER TABLE demo.${dbName}.${topTable} ADD COLUMN victim BIGINT; + INSERT INTO demo.${dbName}.${topTable} + (id, MixedName, metric, added, added_second, victim) + VALUES (5, 'epsilon', 50, 'added-v5', 500, 5000); + """ + String topCpReadd = latestSnapshotId(topTable) + sql """ALTER TABLE `${catalogName}`.`${dbName}`.`${topTable}` CREATE TAG top_cp_readd""" + Thread.sleep(1100) + + // Scenario S08 x T00/T01/T02/T05: compatible INT -> BIGINT promotion. + spark_iceberg_multi """ + ALTER TABLE demo.${dbName}.${topTable} ALTER COLUMN metric TYPE BIGINT; + INSERT INTO demo.${dbName}.${topTable} + (id, MixedName, metric, added, added_second, victim) + VALUES (6, 'zeta', 6000000000, 'added-v6', 600, 6000); + """ + String topCpPromote = latestSnapshotId(topTable) + sql """ALTER TABLE `${catalogName}`.`${dbName}`.`${topTable}` CREATE TAG top_cp_promote""" + + spark_iceberg_multi """ + DROP TABLE IF EXISTS demo.${dbName}.${nestedTable}; + CREATE TABLE demo.${dbName}.${nestedTable} ( + id INT, + payload STRUCT, + attributes MAP>, + events ARRAY> + ) USING iceberg + TBLPROPERTIES ('format-version'='2', 'write.format.default'='orc'); + INSERT INTO demo.${dbName}.${nestedTable} VALUES ( + 1, + named_struct('old_child', 10, 'keep', 11), + map('a', named_struct('old_child', 20, 'keep', 21)), + array(named_struct('old_child', 30, 'keep', 31)) + ); + """ + String nestedCp0 = latestSnapshotId(nestedTable) + sql """ALTER TABLE `${catalogName}`.`${dbName}`.`${nestedTable}` CREATE TAG nested_cp0""" + + // Scenario S09/S14/S15 x T00/T01/T02/T05: + // add children to STRUCT, MAP value struct and ARRAY element struct. + spark_iceberg_multi """ + ALTER TABLE demo.${dbName}.${nestedTable} ADD COLUMN payload.added_child INT; + ALTER TABLE demo.${dbName}.${nestedTable} ADD COLUMN attributes.value.added_child INT; + ALTER TABLE demo.${dbName}.${nestedTable} ADD COLUMN events.element.added_child INT; + INSERT INTO demo.${dbName}.${nestedTable} VALUES ( + 2, + named_struct('old_child', 110, 'keep', 111, 'added_child', 112), + map('a', named_struct('old_child', 120, 'keep', 121, 'added_child', 122)), + array(named_struct('old_child', 130, 'keep', 131, 'added_child', 132)) + ); + """ + String nestedCpAdd = latestSnapshotId(nestedTable) + + // Scenario S10/S14/S15 x T00/T01/T02/T05: + // rename nested children and verify both positive and negative binding. + spark_iceberg_multi """ + ALTER TABLE demo.${dbName}.${nestedTable} + RENAME COLUMN payload.old_child TO renamed_child; + ALTER TABLE demo.${dbName}.${nestedTable} + RENAME COLUMN attributes.value.old_child TO renamed_child; + ALTER TABLE demo.${dbName}.${nestedTable} + RENAME COLUMN events.element.old_child TO renamed_child; + INSERT INTO demo.${dbName}.${nestedTable} VALUES ( + 3, + named_struct('renamed_child', 210, 'keep', 211, 'added_child', 212), + map('a', named_struct('renamed_child', 220, 'keep', 221, 'added_child', 222)), + array(named_struct('renamed_child', 230, 'keep', 231, 'added_child', 232)) + ); + """ + String nestedCpRename = latestSnapshotId(nestedTable) + sql """ALTER TABLE `${catalogName}`.`${dbName}`.`${nestedTable}` CREATE TAG nested_cp_rename""" + + // Scenario S11/S12/S13 x T00/T01/T02/T05: + // drop/re-add a nested name with a new ID and promote the surviving child type. + spark_iceberg_multi """ + ALTER TABLE demo.${dbName}.${nestedTable} DROP COLUMN payload.renamed_child; + ALTER TABLE demo.${dbName}.${nestedTable} DROP COLUMN attributes.value.renamed_child; + ALTER TABLE demo.${dbName}.${nestedTable} DROP COLUMN events.element.renamed_child; + ALTER TABLE demo.${dbName}.${nestedTable} ADD COLUMN payload.renamed_child BIGINT; + ALTER TABLE demo.${dbName}.${nestedTable} ADD COLUMN attributes.value.renamed_child BIGINT; + ALTER TABLE demo.${dbName}.${nestedTable} ADD COLUMN events.element.renamed_child BIGINT; + ALTER TABLE demo.${dbName}.${nestedTable} ALTER COLUMN payload.keep TYPE BIGINT; + INSERT INTO demo.${dbName}.${nestedTable} VALUES ( + 4, + named_struct('keep', 311, 'added_child', 312, 'renamed_child', 3100), + map('a', named_struct('keep', 321, 'added_child', 322, 'renamed_child', 3200)), + array(named_struct('keep', 331, 'added_child', 332, 'renamed_child', 3300)) + ); + """ + String nestedCpReadd = latestSnapshotId(nestedTable) + + spark_iceberg_multi """ + SET spark.sql.shuffle.partitions=1; + DROP TABLE IF EXISTS demo.${dbName}.${deleteTable}; + CREATE TABLE demo.${dbName}.${deleteTable} ( + id INT, + old_name STRING, + category STRING, + event_time TIMESTAMP + ) USING iceberg + PARTITIONED BY (days(event_time)) + TBLPROPERTIES ( + 'format-version'='2', + 'write.delete.mode'='merge-on-read', + 'write.update.mode'='merge-on-read', + 'write.merge.mode'='merge-on-read', + 'write.distribution-mode'='none', + 'write.target-file-size-bytes'='134217728' + ); + INSERT INTO demo.${dbName}.${deleteTable} + SELECT /*+ COALESCE(1) */ id, old_name, category, event_time FROM VALUES + (1, 'a', 'x', TIMESTAMP '2026-01-01 01:00:00'), + (2, 'b', 'x', TIMESTAMP '2026-01-01 02:00:00'), + (3, 'c', 'y', TIMESTAMP '2026-01-02 01:00:00') + AS t(id, old_name, category, event_time); + """ + String deleteCp0 = latestSnapshotId(deleteTable) + sql """ALTER TABLE `${catalogName}`.`${dbName}`.`${deleteTable}` CREATE TAG delete_cp0""" + + // Scenario S04/S17 x TC03/TC04: + // position deletes after rename and partition-spec evolution must not affect the old ref. + spark_iceberg_multi """ + ALTER TABLE demo.${dbName}.${deleteTable} RENAME COLUMN old_name TO new_name; + ALTER TABLE demo.${dbName}.${deleteTable} ADD PARTITION FIELD bucket(2, id); + INSERT INTO demo.${dbName}.${deleteTable} VALUES + (4, 'd', 'y', TIMESTAMP '2026-01-02 02:00:00'); + DELETE FROM demo.${dbName}.${deleteTable} WHERE id = 2; + """ + String deleteCpAfter = latestSnapshotId(deleteTable) + + sql """switch ${catalogName}""" + sql """use ${dbName}""" + sql """refresh catalog ${catalogName}""" + + // Scenario S09-S15 x T00-T13 x TC02/TC04: + // exercise the nested Iceberg DDL implemented on the latest master through Doris itself, + // then combine it with snapshots, tags, a branch, dual-snapshot binding and a delete. + sql """drop table if exists ${dorisNestedTable}""" + sql """ + create table ${dorisNestedTable} ( + id int, + info struct, + events array>, + attrs map> + ) + """ + sql """ + insert into ${dorisNestedTable} values ( + 1, + struct(10, 'old-info'), + array(struct(100)), + map('k', struct(1000)) + ) + """ + String dorisNestedCp0 = sql(""" + select snapshot_id + from ${dorisNestedTable}\$snapshots + order by committed_at desc + limit 1 + """)[0][0].toString() + sql """ + alter table ${dorisNestedTable} + create tag doris_nested_cp0 as of version ${dorisNestedCp0} + """ + sql """ + alter table ${dorisNestedTable} + create branch doris_nested_cp0_branch as of version ${dorisNestedCp0} + """ + Thread.sleep(1100) + + // Scenario S09/S14/S15: Doris adds fields at nested STRUCT, ARRAY element and MAP value paths. + sql """ + alter table ${dorisNestedTable} + add column info.added string null after metric + """ + sql """ + alter table ${dorisNestedTable} + add column events.element.added int null after score + """ + sql """ + alter table ${dorisNestedTable} + add column attrs.value.added int null after code + """ + sql """ + insert into ${dorisNestedTable} values ( + 2, + struct(20, 'info-added', 'after-add'), + array(struct(200, 201)), + map('k', struct(2000, 2001)) + ) + """ + String dorisNestedCpAdd = sql(""" + select snapshot_id + from ${dorisNestedTable}\$snapshots + order by committed_at desc + limit 1 + """)[0][0].toString() + sql """ + alter table ${dorisNestedTable} + create tag doris_nested_cp_add as of version ${dorisNestedCpAdd} + """ + Thread.sleep(1100) + + // Scenario S10/S13/S14/S15: rename nested fields and promote sibling field types in one timeline. + sql """alter table ${dorisNestedTable} rename column info.added to renamed""" + sql """ + alter table ${dorisNestedTable} + rename column events.element.added to renamed + """ + sql """ + alter table ${dorisNestedTable} + rename column attrs.value.added to renamed + """ + sql """alter table ${dorisNestedTable} modify column info.metric bigint""" + sql """ + alter table ${dorisNestedTable} + modify column events.element.score bigint + """ + sql """ + alter table ${dorisNestedTable} + modify column attrs.value.code bigint + """ + sql """ + insert into ${dorisNestedTable} values ( + 3, + struct(cast(30 as bigint), 'info-renamed', 'after-rename'), + array(struct(cast(300 as bigint), 301)), + map('k', struct(cast(3000 as bigint), 3001)) + ) + """ + String dorisNestedCpRename = sql(""" + select snapshot_id + from ${dorisNestedTable}\$snapshots + order by committed_at desc + limit 1 + """)[0][0].toString() + sql """ + alter table ${dorisNestedTable} + create tag doris_nested_cp_rename as of version ${dorisNestedCpRename} + """ + Thread.sleep(1100) + + // Scenario S11/S12/S14/S15: re-add the same nested names with new BIGINT field IDs. + sql """alter table ${dorisNestedTable} drop column info.renamed""" + sql """alter table ${dorisNestedTable} drop column events.element.renamed""" + sql """alter table ${dorisNestedTable} drop column attrs.value.renamed""" + sql """alter table ${dorisNestedTable} add column info.renamed bigint null""" + sql """ + alter table ${dorisNestedTable} + add column events.element.renamed bigint null + """ + sql """ + alter table ${dorisNestedTable} + add column attrs.value.renamed bigint null + """ + sql """ + insert into ${dorisNestedTable} values ( + 4, + struct(cast(40 as bigint), 'info-readd', cast(4001 as bigint)), + array(struct(cast(400 as bigint), cast(401 as bigint))), + map('k', struct(cast(4000 as bigint), cast(4001 as bigint))) + ) + """ + String dorisNestedCpReadd = sql(""" + select snapshot_id + from ${dorisNestedTable}\$snapshots + order by committed_at desc + limit 1 + """)[0][0].toString() + sql """ + alter table ${dorisNestedTable} + create tag doris_nested_cp_readd as of version ${dorisNestedCpReadd} + """ + + // Scenario TC04: a delete after nested evolution must not change any older snapshot/tag. + sql """delete from ${dorisNestedTable} where id = 2""" + String dorisNestedCpDelete = sql(""" + select snapshot_id + from ${dorisNestedTable}\$snapshots + order by committed_at desc + limit 1 + """)[0][0].toString() + + // Scenario TC02/T01/T05/T06/T08: every old reference exposes its own nested schema. + assertEquals([[1, 10, 100, 1000]], + sql(""" + select id, info.metric, events[1].score, attrs['k'].code + from ${dorisNestedTable} for version as of ${dorisNestedCp0} + order by id + """)) + assertEquals([[1, 10, 100, 1000]], + sql(""" + select id, info.metric, events[1].score, attrs['k'].code + from ${dorisNestedTable} for version as of 'doris_nested_cp0' + order by id + """)) + assertEquals([[1, 10, 100, 1000]], + sql(""" + select id, info.metric, events[1].score, attrs['k'].code + from ${dorisNestedTable}@tag(doris_nested_cp0) + order by id + """)) + // Negative contract: an old branch currently leaks the latest BIGINT nested types. + assertEquals([[1, 10L, 100L, 1000L]], + sql(""" + select id, info.metric, events[1].score, attrs['k'].code + from ${dorisNestedTable}@branch(doris_nested_cp0_branch) + order by id + """)) + assertUnknownColumn(""" + select info.renamed + from ${dorisNestedTable} for version as of ${dorisNestedCp0} + """, "renamed") + + // Scenario TC02/T03/T04: complex-field time travel uses the pre-change nested schema. + List> dorisNestedCp0Time = sql(""" + select date_format(date_add(committed_at, interval 1 second), '%Y-%m-%d %H:%i:%s'), + cast(unix_timestamp(committed_at) * 1000 + 999 as bigint) + from ${dorisNestedTable}\$snapshots + where snapshot_id = ${dorisNestedCp0} + """) + assertEquals([[1, 10]], + sql(""" + select id, info.metric + from ${dorisNestedTable} + for time as of "${dorisNestedCp0Time[0][0]}" + order by id + """)) + // Scenario TC03-negative: Iceberg accepts time strings, not Paimon-style epoch millis. + test { + sql """ + select id, info.metric + from ${dorisNestedTable} + for time as of ${dorisNestedCp0Time[0][1]} + order by id + """ + exception "can't parse time" + } + + // Scenario TC02: nested add/rename/drop-readd checkpoints verify projection and predicates. + assertEquals([[1, null, null, null], [2, "info-added", 201, 2001]], + sql(""" + select id, info.added, events[1].added, attrs['k'].added + from ${dorisNestedTable} for version as of ${dorisNestedCpAdd} + order by id + """)) + assertEquals([[1, null, null, null], [2, "info-added", 201, 2001], + [3, "info-renamed", 301, 3001]], + sql(""" + select id, info.renamed, events[1].renamed, attrs['k'].renamed + from ${dorisNestedTable} for version as of ${dorisNestedCpRename} + where info.metric >= 10 + order by id + """)) + assertUnknownColumn(""" + select info.added + from ${dorisNestedTable} for version as of ${dorisNestedCpRename} + """, "added") + assertEquals([[1, null, null, null], [2, null, null, null], + [3, null, null, null], [4, 4001L, 401L, 4001L]], + sql(""" + select id, info.renamed, events[1].renamed, attrs['k'].renamed + from ${dorisNestedTable} for version as of ${dorisNestedCpReadd} + order by id + """)) + + // Scenario TC04: delete is visible only at/after the delete snapshot. + assertEquals([1, 2, 3, 4], + sql(""" + select id + from ${dorisNestedTable} for version as of ${dorisNestedCpReadd} + order by id + """).collect { it[0] }) + assertEquals([1, 3, 4], + sql(""" + select id + from ${dorisNestedTable} for version as of ${dorisNestedCpDelete} + order by id + """).collect { it[0] }) + + // Scenario TC01: current schema uses only the new names and promoted/re-added types. + List currentColumns = sql("""desc ${topTable}""").collect { it[0].toString() } + assertEquals(["added_second", "id", "MixedName", "added", "metric", "victim"], currentColumns) + assertEquals([[1, null], [2, null], [3, null], [4, null], [5, 5000L], [6, 6000L]], + sql("""select id, victim from ${topTable} order by id""")) + assertEquals([[6, 6000000000L]], + sql("""select id, metric from ${topTable} where metric > 5000000000 order by id""")) + assertUnknownColumn("""select old_name from ${topTable}""", "old_name") + + // Scenario T01/T05/T06/T08: the pre-change snapshot/tag/branch binds old_name and victim. + List> topCp0Rows = [[1, "alpha", "old-v1", 10]] + assertEquals(topCp0Rows, sql(""" + select id, old_name, victim, metric + from ${topTable} for version as of ${topCp0} + order by id + """)) + assertEquals(topCp0Rows, sql(""" + select id, old_name, victim, metric + from ${topTable} for version as of 'top_cp0' + order by id + """)) + assertEquals(topCp0Rows, sql(""" + select id, old_name, victim, metric + from ${topTable}@tag(top_cp0) + order by id + """)) + // Negative contract: an old branch is currently analyzed with the latest rename schema. + test { + sql """ + select id, old_name, victim, metric + from ${topTable}@branch(top_cp0_branch) + order by id + """ + exception "Unknown column 'old_name'" + } + assertUnknownColumn(""" + select MixedName from ${topTable} for version as of ${topCp0} + """, "MixedName") + + // Scenario T03/T04: validate string time travel and reject unsupported epoch millis. + List> cp0TimeRows = sql(""" + select date_format(date_add(committed_at, interval 1 second), '%Y-%m-%d %H:%i:%s'), + cast(unix_timestamp(committed_at) * 1000 + 999 as bigint) + from ${topTable}\$snapshots + where snapshot_id = ${topCp0} + """) + assertEquals(1, cp0TimeRows.size()) + assertEquals(topCp0Rows, sql(""" + select id, old_name, victim, metric + from ${topTable} for time as of "${cp0TimeRows[0][0]}" + order by id + """)) + test { + sql """ + select id, old_name, victim, metric + from ${topTable} for time as of ${cp0TimeRows[0][1]} + order by id + """ + exception "can't parse time" + } + + // Scenario S01-S05: every checkpoint verifies projection, predicate and aggregation. + assertEquals([[1, "alpha", null], [2, "beta", "added-v2"]], + sql(""" + select id, old_name, added + from ${topTable} for version as of ${topCpAdd} + where metric >= 10 + order by id + """)) + assertEquals([[3L, 60L]], + sql(""" + select count(*), sum(metric) + from ${topTable} for version as of ${topCpRename} + """)) + assertUnknownColumn(""" + select old_name from ${topTable} for version as of ${topCpRename} + """, "old_name") + assertEquals([[1, "old-v1"], [2, "old-v2"], [3, "old-v3"]], + sql(""" + select id, victim + from ${topTable} for version as of 'top_cp_rename' + where victim like 'old-%' + order by id + """)) + assertUnknownColumn(""" + select victim from ${topTable} for version as of ${topCpDrop} + """, "victim") + assertEquals([[1, null], [2, null], [3, null], [4, null], [5, 5000L]], + sql(""" + select id, victim + from ${topTable} for version as of ${topCpReadd} + order by id + """)) + assertEquals([[6, 6000000000L]], + sql(""" + select id, metric + from ${topTable} for version as of ${topCpPromote} + where metric > 5000000000 + """)) + + // Scenario TC02: nested projection/predicate use each snapshot's nested field IDs. + assertEquals([[1, 10, 20, 30]], + sql(""" + select id, payload.old_child, + element_at(attributes, 'a').old_child, + events[1].old_child + from ${nestedTable} for version as of ${nestedCp0} + where payload.old_child = 10 + """)) + assertEquals([[1, null, null, null], [2, 112, 122, 132]], + sql(""" + select id, payload.added_child, + element_at(attributes, 'a').added_child, + events[1].added_child + from ${nestedTable} for version as of ${nestedCpAdd} + order by id + """)) + assertEquals([[1, 10, 20, 30], [2, 110, 120, 130], [3, 210, 220, 230]], + sql(""" + select id, payload.renamed_child, + element_at(attributes, 'a').renamed_child, + events[1].renamed_child + from ${nestedTable} for version as of 'nested_cp_rename' + order by id + """)) + assertUnknownColumn(""" + select payload.old_child + from ${nestedTable} for version as of ${nestedCpRename} + """, "old_child") + assertEquals([[1, null], [2, null], [3, null], [4, 3100L]], + sql(""" + select id, payload.renamed_child + from ${nestedTable} for version as of ${nestedCpReadd} + order by id + """)) + + // Scenario TC03/TC04: delete visibility and renamed delete-table fields are snapshot-local. + assertEquals([[1, "a"], [2, "b"], [3, "c"]], + sql(""" + select id, old_name + from ${deleteTable} for version as of 'delete_cp0' + order by id + """)) + assertEquals([[1, "a"], [3, "c"], [4, "d"]], + sql("""select id, new_name from ${deleteTable} order by id""")) + assertEquals([[1, "a"], [3, "c"], [4, "d"]], + sql(""" + select id, new_name + from ${deleteTable} for version as of ${deleteCpAfter} + order by id + """)) + assertTrue(((Number) sql(""" + select count(*) from ${deleteTable}\$position_deletes + """)[0][0]).longValue() > 0L) + + // Scenario TC08/S20: illegal narrowing and dropping a partition source are atomic failures. + test { + sql """alter table ${topTable} modify column metric int""" + exception "Cannot" + } + test { + sql """alter table ${deleteTable} drop column event_time""" + exception "Cannot" + } + assertEquals([[6, 6000000000L]], + sql("""select id, metric from ${topTable} where id = 6""")) + assertEquals([[1, "a"], [3, "c"], [4, "d"]], + sql("""select id, new_name from ${deleteTable} order by id""")) + + // Scenario TC09/R13/R17: cache-off and scanner V1/V2 must produce identical historical rows. + sql """switch ${noCacheCatalogName}""" + sql """use ${dbName}""" + List> noCacheRows = sql(""" + select id, old_name, victim, metric + from ${topTable} for version as of ${topCp0} + order by id + """) + assertEquals(topCp0Rows, noCacheRows) + sql """switch ${catalogName}""" + sql """use ${dbName}""" + sql """set enable_file_scanner_v2=false""" + List> legacyRows = sql(""" + select id, old_name, victim, metric + from ${topTable} for version as of ${topCp0} + order by id + """) + sql """set enable_file_scanner_v2=true""" + List> v2Rows = sql(""" + select id, old_name, victim, metric + from ${topTable} for version as of ${topCp0} + order by id + """) + assertEquals(legacyRows, v2Rows) + + // Scenario T11: an unknown snapshot/tag must fail instead of silently reading latest. + test { + sql """select * from ${topTable} for version as of 9223372036854775807""" + exception "does not have snapshotId 9223372036854775807" + } + test { + sql """select * from ${topTable} for version as of 'missing_schema_tag'""" + exception "does not have tag or branch named missing_schema_tag" + } + } finally { + sql """set enable_file_scanner_v2=false""" + sql """drop catalog if exists ${catalogName}""" + sql """drop catalog if exists ${noCacheCatalogName}""" + } +} diff --git a/regression-test/suites/external_table_p0/iceberg/write/ICEBERG_WRITE_P0_COVERAGE.md b/regression-test/suites/external_table_p0/iceberg/write/ICEBERG_WRITE_P0_COVERAGE.md new file mode 100644 index 00000000000000..7f0ba5b849c725 --- /dev/null +++ b/regression-test/suites/external_table_p0/iceberg/write/ICEBERG_WRITE_P0_COVERAGE.md @@ -0,0 +1,120 @@ + + +# Iceberg 写入 P0 覆盖矩阵 + +## 范围与判定原则 + +本文档覆盖 Doris 向 Iceberg 表写入时的正确性、兼容性和失败原子性。矩阵既检查单项能力，也检查 schema change、Partition Evolution、snapshot/tag/branch、行级 delete/update/merge、表模型、分区与 bucket、数据类型和 NULL 语义之间的交互。 + +所有正向 suite 都在最后一次写入后比较 Doris 与 Spark 的同表逻辑结果；涉及 transform metadata 的 suite 还比较物理分区值，涉及历史引用时另行校验 snapshot、tag 和 branch 的隔离性。 + +覆盖状态含义： + +- 已覆盖（验证通过）：P0 suite 对该场景有确定性结果断言，且已在多 BE 环境验证。 +- 预期拒绝：Doris 明确不支持该操作，P0 suite 验证错误信息与失败原子性。 +- 已覆盖（隔离负向）：已形成可复现产品问题的 regression；默认 P0 隔离运行，避免杀死共享 BE 或提交不可读文件。 + +## 风险点 + +| 编号 | 风险描述 | 来源 | 影响面 | 级别 | +| --- | --- | --- | --- | --- | +| R01 | schema change 后 writer 仍按旧列位置或旧 field id 写入，造成静默错列 | 白盒：Iceberg field id 与 Doris slot 映射 | 数据正确性 | P0 | +| R02 | Partition Evolution 后新文件落入旧 spec、分区值计算错误，或跨 spec 过滤漏数 | 黑盒 + 白盒：多 partition spec 并存 | 写入与查询正确性 | P0 | +| R03 | 字符串、数值、日期时间、decimal、布尔和 NULL 作为 identity/bucket/truncate/time transform 源时行为不一致 | 黑盒：类型与边界输入 | 分区路由、裁剪 | P0 | +| R04 | schema/partition 演进后 snapshot、tag、branch 绑定了错误 schema 或数据版本 | 黑盒：历史读与引用 | time travel 正确性 | P0 | +| R05 | MOR 的 delete/update/merge 跨新旧 spec 时生成错误 delete file；COW 被拒绝后仍发布快照 | 白盒：row-level DML commit | 数据丢失、失败原子性 | P0 | +| R06 | Duplicate、Unique MOW、Unique MOR、Aggregate 源表语义在 INSERT SELECT 时被改变 | 黑盒：不同 Doris 表模型 | 跨表写入正确性 | P0 | +| R07 | RANGE/LIST/无分区源表以及 HASH/RANDOM/AUTO bucket 在多 BE 执行时产生重复或丢行 | 黑盒 + 白盒：分布式 exchange 与 sink writer | 分布式写入正确性 | P0 | +| R08 | primitive、ARRAY、MAP、STRUCT 及嵌套 NULL 在 schema change 前后写入错误 | 黑盒：复杂类型与 NULL | 数据正确性、兼容性 | P0 | +| R09 | NULL 写入 Iceberg required 列未报错、部分数据或空快照被提交 | 白盒：required 校验与 commit | 约束、失败原子性 | P0 | +| R10 | INSERT OVERWRITE 在演进后的当前 spec、branch 或 NULL 分区上误删其他分区 | 黑盒：覆盖写 | 数据丢失 | P0 | +| R11 | 单 BE 可通过但多 BE 并发 sink 出现文件名、commit 或分区冲突 | 白盒：并行 writer 与统一 commit | 分布式稳定性 | P0 | +| R12 | nullable STRING 或 DML 产生的 Nullable block 经过 truncate transform 时 BE FATAL | 白盒：partition transformer 列类型约束 | 集群可用性 | P0 | +| R13 | MERGE 的多个源行匹配同一目标行时未执行基数校验，错误提交重复数据 | 黑盒 + 白盒：MERGE cardinality 与 commit | 数据正确性 | P0 | +| R14 | branch 写入污染 main，或 tag/不支持的 branch 行级 DML 失败后仍发布快照 | 黑盒：reference write 边界 | 历史引用、失败原子性 | P0 | +| R15 | 当前 spec 覆盖写未正确清理旧 spec delete file，或失败覆盖写留下部分快照 | 白盒：overwrite commit 与 delete file | 数据丢失、失败原子性 | P0 | +| R16 | CTAS 对复杂类型、NULL、分区 transform、文件格式和失败清理的行为不一致 | 黑盒：DDL + writer 一体提交 | schema、文件格式、原子性 | P0 | +| R17 | sort order、distribution mode、多次文件 flush 和并发 commit 组合导致乱序、丢行或重复提交 | 白盒：exchange、sort writer、optimistic commit | 分布式正确性、稳定性 | P0 | +| R18 | STRING identity/bucket/truncate 对空串、中文、emoji、组合字符和 NULL 的物理分区值计算错误 | 黑盒：UTF-8 transform metadata | 分区路由、裁剪 | P0 | + +## 组合覆盖 + +| 维度 | 场景 | 状态 | P0 suite | +| --- | --- | --- | --- | +| 基础写入 | Parquet/ORC、primitive/复杂类型、INSERT/OVERWRITE | 已覆盖 | `test_iceberg_write_insert`、`test_iceberg_insert_overwrite` | +| Partition transform | identity、bucket、truncate、year/month/day/hour | 已覆盖 | `test_iceberg_write_transform_partitions`、`test_iceberg_static_partition_overwrite` | +| schema + partition 演进 | add/rename/drop/type promotion 与 ADD/REPLACE/DROP partition field 后继续写入和过滤 | 已覆盖（验证通过） | `test_iceberg_write_evolution_refs` | +| 复杂类型演进 | ARRAY/MAP/STRUCT promotion、STRUCT 新增字段、旧文件与新写入并存 | 已覆盖（验证通过） | `test_iceberg_write_complex_evolution` | +| 历史版本 | 演进前后 snapshot、tag、branch；branch 独立写入和覆盖写 | 已覆盖（验证通过） | `test_iceberg_write_evolution_refs` | +| MOR | partition evolution 后 DELETE/UPDATE/MERGE，校验当前、delete files 与历史版本 | 已覆盖（验证通过） | `test_iceberg_write_dml_modes_evolution` | +| COW | partition evolution 后 DELETE/UPDATE/MERGE 拒绝，且数据和 snapshot 数不变 | 预期拒绝 | `test_iceberg_write_dml_modes_evolution` | +| Doris 源表模型 | Duplicate、Unique MOW、Unique MOR、Aggregate | 已覆盖（验证通过） | `test_iceberg_write_source_models` | +| Doris 源分区 | 无分区、RANGE、LIST | 已覆盖（验证通过） | `test_iceberg_write_source_models` | +| Doris 源 bucket | HASH 固定 bucket、RANDOM bucket、HASH AUTO bucket | 已覆盖（验证通过） | `test_iceberg_write_source_models` | +| 分区源类型 | STRING/INT/BIGINT/DATE/DATETIME/DECIMAL 的 bucket 与适用 transform；BOOLEAN identity 与非法 bucket | 已覆盖（验证通过） | `test_iceberg_write_partition_types_null` | +| NULL 分区 | identity NULL、数值/decimal bucket 与 truncate NULL、time transform NULL、多列组合 NULL | 已覆盖（验证通过） | `test_iceberg_write_partition_types_null` | +| nullable STRING truncate | nullable STRING 经过 truncate transform 的 INSERT，以及 UPDATE 产生的 Nullable projection 写入 | 已覆盖（隔离负向） | `test_iceberg_write_nullable_truncate_negative` | +| MERGE 完整语义 | 条件 MATCHED、DELETE/UPDATE、多个条件 NOT MATCHED、NULL-safe 与普通 NULL key | 已覆盖（验证通过） | `test_iceberg_write_merge_semantics` | +| MERGE 基数约束 | 多个源行匹配同一目标行必须整句失败且不发布快照 | 已覆盖（隔离负向） | `test_iceberg_write_merge_duplicate_source_negative` | +| MERGE + STRING truncate | required truncate 源列经 MERGE nullable projection 写入 | 已覆盖（隔离负向） | `test_iceberg_write_merge_truncate_negative` | +| branch/tag 写入边界 | branch INSERT/OVERWRITE 隔离；tag 写入和 branch DELETE/UPDATE/MERGE 明确拒绝 | 已覆盖（验证通过） | `test_iceberg_write_branch_dml_boundary` | +| nullable 数据 | 顶层 NULL、ARRAY NULL 元素、MAP NULL value、STRUCT NULL child | 已覆盖并增强 | `test_iceberg_write_insert`、`test_iceberg_write_complex_evolution` | +| required 列正向与 schema change | required 列合法写入、nullable 列写 NULL、增加 required 列与 nullable→required 拒绝 | 已覆盖（验证通过） | `test_iceberg_write_nullability_atomicity` | +| required 列写 NULL | VALUES 与分布式 INSERT SELECT 混合批次写 NULL | 已覆盖（隔离负向） | `test_iceberg_write_required_null_values_negative`、`test_iceberg_write_required_null_select_negative` | +| 覆盖写 | 当前 spec、静态分区、branch、空输入、连续多次 partition evolution、NULL 当前分区 | 已覆盖并增强 | `test_iceberg_static_partition_overwrite`、`test_iceberg_write_evolution_refs`、`test_iceberg_write_overwrite_evolution` | +| 覆盖写 + delete files | MOR DELETE/UPDATE/MERGE 后覆盖写，演进前后 delete files 与历史 tag 共存 | 已覆盖（验证通过） | `test_iceberg_write_overwrite_delete_files` | +| 覆盖写失败原子性 | main/branch 分布式严格类型转换失败、快照/文件/数据不变、修正后重试 | 已覆盖（验证通过） | `test_iceberg_write_overwrite_atomicity` | +| STRING 物理 transform | identity、nullable bucket、required truncate 的 UTF-8 边界值及 transform width evolution | 已覆盖（验证通过） | `test_iceberg_write_string_transform_metadata` | +| CTAS | 复杂类型、嵌套 NULL、identity+bucket、ORC 压缩、失败建表清理 | 已覆盖（验证通过） | `test_iceberg_write_ctas_format_boundary` | +| 文件格式边界 | Parquet/ORC 正向写入；Avro 表写入明确拒绝并保持快照和文件不变 | 已覆盖（正向 + 预期拒绝） | `test_iceberg_write_ctas_format_boundary` | +| 排序与分布属性 | 多列 sort order、NULL ordering、none/hash/range distribution、强制多文件 flush | 已覆盖（验证通过） | `test_iceberg_write_order_distribution_properties` | +| 并发写入 | 同行冲突 MERGE 的串行化不变量、非冲突分布式 append | 已覆盖（验证通过） | `test_iceberg_write_concurrent_merge_invariants` | +| 分布式执行 | 多 bucket 源表、多分区 Iceberg sink、多 BE writer、suite 间无共享 catalog/database | 已覆盖（验证通过） | 所有本次新增 suite | +| Spark 交叉验证 | 十五个正向 suite 在最后一次写入后逐行比较；STRING transform suite 另行比较演进前后的物理分区 metadata | 已覆盖（验证通过） | 十五个正向 suite；物理 metadata 见 `test_iceberg_write_string_transform_metadata` | + +## 本次新增用例设计 + +| 用例 | 目标 | 覆盖风险 | 测试维度 | 前置条件 | 负载描述 | 执行预期 | +| --- | --- | --- | --- | --- | --- | --- | +| W01 | 验证 schema 与 partition spec 同时演进后的写入、过滤和历史引用 | R01、R02、R04、R10 | 功能、正确性、兼容性 | Iceberg REST catalog | 演进前后多批 Doris 写入，建立 snapshot/tag/branch，并对 branch 覆盖写 | 当前、历史和 branch 各自返回确定数据；跨 spec 过滤不漏数 | +| W02 | 验证复杂类型 field id 在演进后保持正确 | R01、R08 | 功能、正确性 | Iceberg v2 | ARRAY/MAP value promotion、STRUCT child promotion/add，写入含嵌套 NULL 的新旧行 | 旧值按新 schema 可读，新值不串字段，嵌套 NULL 保留 | +| W03 | 验证 MOR/COW 与 partition evolution、NULL 分区、time travel 的交互 | R02、R04、R05 | 功能、正确性、异常 | Iceberg v2 MOR/COW | MOR 执行 delete/update/merge；COW 执行相同操作 | MOR 当前与历史版本一致；COW 明确拒绝且无新 snapshot | +| W04 | 验证不同 Doris 表模型、分区和 bucket 作为 Iceberg 写入源 | R06、R07、R11 | 正确性、兼容性 | 多 BE Doris | 四种表模型、三种分区方式、HASH/RANDOM/AUTO bucket 执行 INSERT SELECT | 写入结果保持各源表语义，无重复或丢行 | +| W05 | 验证不同类型与 NULL 的 partition/bucket transform | R02、R03、R11 | 功能、正确性、边界 | Iceberg v2 | identity/bucket/truncate/time transform 多列组合，包含 NULL | 数据与 `$partitions` 统计一致；NULL 行可过滤且可继续写入 | +| W06 | 验证 required/nullable schema change 与合法写入 | R09、R11 | 异常、正确性 | Iceberg required 列 | 拒绝增加无默认值 required 列和 nullable→required；执行 VALUES/INSERT SELECT 合法写入 | schema change 失败不产生 snapshot；合法写入与 Spark 结果一致 | +| W07 | 验证 required 列 NULL 拒绝和 statement 原子性 | R09、R11 | 隔离负向、正确性 | 隔离 Iceberg database | VALUES 写 NULL；多 bucket 源表 INSERT SELECT 混合有效与 NULL 行 | 修复前会错误提交并产生不可读文件；修复后整条语句在 snapshot 发布前拒绝 | +| W08 | 验证 STRING truncate 的 Nullable block 处理 | R03、R05、R12 | 隔离负向、稳定性 | 可重启的隔离 Doris 集群 | nullable STRING INSERT；partition evolution 后 UPDATE 产生 Nullable block | 修复前 BE FATAL；修复后写入成功并保持 NULL 分区语义 | +| W09 | 验证 MERGE 条件动作、多个 NOT MATCHED 与 NULL key 语义 | R02、R03、R05 | 功能、正确性 | Iceberg v2 MOR | identity/bucket 分区间移动、删除、插入、NULL-safe 与普通等值匹配 | 每个源行只选择一个动作，Spark 与 Doris 结果一致 | +| W10 | 验证 MERGE 多源匹配单目标的基数约束 | R13 | 隔离负向、原子性 | Iceberg v2 MOR | 两个源行同时更新一个目标行 | 修复前错误提交重复行；修复后整句拒绝且无新快照和文件 | +| W11 | 验证 branch/tag 的写入能力边界 | R04、R14 | 功能、异常、原子性 | 已建立 branch 与 tag | branch INSERT/OVERWRITE；branch 行级 DML 与 tag 写入 | branch 与 main 隔离；不支持操作明确拒绝且引用不变化 | +| W12 | 验证多次 Partition Evolution 后覆盖写和历史引用 | R02、R04、R10、R15 | 功能、正确性 | Iceberg v2 | ADD/REPLACE/DROP identity、bucket、truncate、day/hour 后动态覆盖写 | 仅替换当前 spec 命中的分区，tag/branch 和旧 spec 保持可读 | +| W13 | 验证 delete files 与覆盖写、演进的交互 | R02、R05、R15 | 正确性、兼容性 | Iceberg v2 MOR | DELETE/UPDATE/MERGE 生成 delete files，再在新旧 spec 上覆盖写 | replacement 行不被旧 delete files 隐藏，历史 tag 不受影响 | +| W14 | 验证 main/branch 覆盖写失败与重试原子性 | R09、R10、R15 | 异常、原子性 | 多 BE Doris | 分布式严格类型转换失败后检查数据、文件和快照，再执行修正重试 | 失败零提交；重试恰好产生一个快照且无重复 | +| W15 | 验证 STRING transform 的真实物理分区值 | R03、R18 | 边界、正确性 | Iceberg v2 | 空串、ASCII、中文、emoji、组合字符、NULL bucket，随后替换 bucket/truncate 宽度 | 行结果和 `$partitions` 物理值均与 Spark 一致 | +| W16 | 验证 CTAS、复杂类型、格式和失败清理 | R08、R09、R16 | 功能、异常、兼容性 | 内部多 bucket 源表 | CTAS 到 ORC 分区表；严格转换失败；向 Avro 表写入 | ORC 与 Spark 一致；失败不遗留表或快照；Avro 明确拒绝 | +| W17 | 验证 sort order、distribution mode 和多文件 flush | R03、R11、R17 | 正确性、稳定性 | 多 BE Doris | NULL sort key、多列升降序、none/hash/range、低 target file size | 计划包含声明排序，多文件总行数正确，三种分布模式结果一致 | +| W18 | 验证并发 MERGE 与 append 的提交不变量 | R11、R13、R17 | 并发、原子性 | 多 BE Doris | readiness barrier 保证两个独立会话同时具备 dispatch 条件；同行更新与互不冲突 append | 同行提交可串行化且基数为一；仅接受 Iceberg validation/commit conflict；非冲突写入无丢失或重复 | +| W19 | 验证 MERGE source projection 进入 truncate transform 的类型安全 | R12、R18 | 隔离负向、稳定性 | 可重启的隔离 Doris 集群 | required STRING truncate 列执行匹配更新与未匹配插入 | 修复前 BE FATAL；修复后 MERGE 成功且物理分区正确 | + +## P0 覆盖检查 + +R01-R18 均映射到至少一个 P0 regression。十五个正向 suite 在最后一次成功写入后均由 Spark/Doris 交叉校验同表逻辑结果；transform、rollover、CTAS property 和失败原子性另有物理 metadata 或状态 oracle。稳定性或已确认正确性缺陷使用独立 suite、完整预期输出和显式隔离开关保存复现，避免默认 P0 破坏共享集群或固化错误结果。 + +本矩阵未覆盖项为 0。COW 行级 DML、branch 行级 DML、tag 写入和 Avro 写入属于当前明确能力边界，均以预期拒绝用例固化错误语义与失败原子性；已确认的产品缺陷均有隔离负向 regression。 diff --git a/regression-test/suites/external_table_p0/iceberg/write/test_iceberg_failed_write_atomicity_retry.groovy b/regression-test/suites/external_table_p0/iceberg/write/test_iceberg_failed_write_atomicity_retry.groovy new file mode 100644 index 00000000000000..8769295d991b4c --- /dev/null +++ b/regression-test/suites/external_table_p0/iceberg/write/test_iceberg_failed_write_atomicity_retry.groovy @@ -0,0 +1,107 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +suite("test_iceberg_failed_write_atomicity_retry", "p0,external") { + String enabled = context.config.otherConfigs.get("enableIcebergTest") + if (enabled == null || !enabled.equalsIgnoreCase("true")) { + logger.info("disable iceberg test.") + return + } + + String catalogName = "test_iceberg_failed_write_atomicity_retry" + String dbName = "failed_write_atomicity_retry_db" + String tableName = "failed_write_atomicity_retry" + String restPort = context.config.otherConfigs.get("iceberg_rest_uri_port") + String minioPort = context.config.otherConfigs.get("iceberg_minio_port") + String externalEnvIp = context.config.otherConfigs.get("externalEnvIp") + + sql """drop catalog if exists ${catalogName}""" + sql """ + create catalog ${catalogName} properties ( + 'type' = 'iceberg', + 'iceberg.catalog.type' = 'rest', + 'uri' = 'http://${externalEnvIp}:${restPort}', + 's3.access_key' = 'admin', + 's3.secret_key' = 'password', + 's3.endpoint' = 'http://${externalEnvIp}:${minioPort}', + 's3.region' = 'us-east-1' + ) + """ + + try { + sql """switch ${catalogName}""" + sql """drop database if exists ${dbName} force""" + sql """create database ${dbName}""" + sql """use ${dbName}""" + + sql """ + create table ${tableName} ( + id int, + payload string + ) + """ + sql """insert into ${tableName} values (1, 'committed')""" + sql """set enable_strict_cast = true""" + + long snapshotsBeforeFailure = (sql """ + select count(*) from ${tableName}\$snapshots + """)[0][0] as long + long filesBeforeFailure = (sql """ + select count(*) from ${tableName}\$files + """)[0][0] as long + + // A pipeline failure after rows reach the Iceberg sink must not publish a partial snapshot. + // Retrying the corrected logical write must therefore make each row visible exactly once. + test { + sql """ + insert into ${tableName} + select cast(if(number = 1, 'invalid-id', cast(number + 2 as string)) as int), + concat('candidate-', number) + from numbers('number' = '3') + """ + exception "can't cast to INT in strict mode" + } + + assertEquals(snapshotsBeforeFailure, (sql """ + select count(*) from ${tableName}\$snapshots + """)[0][0] as long) + assertEquals(filesBeforeFailure, (sql """ + select count(*) from ${tableName}\$files + """)[0][0] as long) + order_qt_failed_write_state """ + select id, payload from ${tableName} order by id + """ + + sql """ + insert into ${tableName} + select number + 2, concat('candidate-', number) + from numbers('number' = '3') + """ + assertEquals(snapshotsBeforeFailure + 1, (sql """ + select count(*) from ${tableName}\$snapshots + """)[0][0] as long) + order_qt_failed_write_retry """ + select id, payload, count(*) + from ${tableName} + group by id, payload + order by id + """ + } finally { + sql """drop database if exists ${catalogName}.${dbName} force""" + sql """drop catalog if exists ${catalogName}""" + } +} diff --git a/regression-test/suites/external_table_p0/iceberg/write/test_iceberg_write_branch_dml_boundary.groovy b/regression-test/suites/external_table_p0/iceberg/write/test_iceberg_write_branch_dml_boundary.groovy new file mode 100644 index 00000000000000..f5f8ef5bb9cd49 --- /dev/null +++ b/regression-test/suites/external_table_p0/iceberg/write/test_iceberg_write_branch_dml_boundary.groovy @@ -0,0 +1,169 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +suite("test_iceberg_write_branch_dml_boundary", + "p0,external,iceberg,external_docker,external_docker_iceberg") { + String enabled = context.config.otherConfigs.get("enableIcebergTest") + if (enabled == null || !enabled.equalsIgnoreCase("true")) { + logger.info("disable iceberg test") + return + } + + String restPort = context.config.otherConfigs.get("iceberg_rest_uri_port") + String minioPort = context.config.otherConfigs.get("iceberg_minio_port") + String externalEnvIp = context.config.otherConfigs.get("externalEnvIp") + String catalogName = "test_iceberg_write_branch_dml_boundary" + String dbName = "iceberg_write_branch_dml_boundary_db" + + sql """drop catalog if exists ${catalogName}""" + sql """ + create catalog ${catalogName} properties ( + "type" = "iceberg", + "iceberg.catalog.type" = "rest", + "uri" = "http://${externalEnvIp}:${restPort}", + "s3.access_key" = "admin", + "s3.secret_key" = "password", + "s3.endpoint" = "http://${externalEnvIp}:${minioPort}", + "s3.region" = "us-east-1" + ) + """ + sql """switch ${catalogName}""" + sql """drop database if exists ${dbName} force""" + sql """create database ${dbName}""" + sql """use ${dbName}""" + + sql """drop table if exists branch_dml_boundary""" + sql """ + create table branch_dml_boundary ( + id int, + region string, + payload string + ) + partition by list (region) () + properties ( + "format-version" = "2", + "write.delete.mode" = "merge-on-read", + "write.update.mode" = "merge-on-read", + "write.merge.mode" = "merge-on-read" + ) + """ + sql """insert into branch_dml_boundary values (1, 'A', 'main')""" + sql """alter table branch_dml_boundary create branch audit_branch""" + sql """alter table branch_dml_boundary create tag protected_tag""" + + // WB01-S01: Seed the exact identity partition that OVERWRITE targets so an + // append-equivalent implementation cannot satisfy the final row oracle. + sql """insert into branch_dml_boundary@branch(audit_branch) values (2, 'B', 'branch-insert')""" + sql """ + insert into branch_dml_boundary@branch(audit_branch) + values (30, 'C', 'branch-overwrite-seed') + """ + assertEquals(1L, (sql """ + select count(*) from branch_dml_boundary@branch(audit_branch) + where payload = 'branch-overwrite-seed' + """)[0][0] as long) + sql """ + insert overwrite table branch_dml_boundary@branch(audit_branch) + values (3, 'C', 'branch-overwrite') + """ + assertEquals(0L, (sql """ + select count(*) from branch_dml_boundary@branch(audit_branch) + where payload = 'branch-overwrite-seed' + """)[0][0] as long) + order_qt_branch_write """ + select id, region, payload + from branch_dml_boundary@branch(audit_branch) + order by id + """ + order_qt_main_after_branch_write """ + select id, region, payload + from branch_dml_boundary + order by id + """ + + long mainSnapshots = (sql """select count(*) from branch_dml_boundary\$snapshots""")[0][0] as long + + // WB01-S02: The current Doris SQL surface does not accept branch-qualified + // targets for row-level DML. Keep the capability boundary explicit and atomic. + test { + sql """delete from branch_dml_boundary@branch(audit_branch) where id = 3""" + exception "@" + } + test { + sql """ + update branch_dml_boundary@branch(audit_branch) + set payload = 'updated' + where id = 3 + """ + exception "@" + } + test { + sql """ + merge into branch_dml_boundary@branch(audit_branch) t + using (select 3 as id, 'merged' as payload) s + on t.id = s.id + when matched then update set payload = s.payload + """ + exception "@" + } + assertEquals(mainSnapshots, + (sql """select count(*) from branch_dml_boundary\$snapshots""")[0][0] as long) + order_qt_branch_after_rejected_dml """ + select id, region, payload + from branch_dml_boundary@branch(audit_branch) + order by id + """ + + // WB01-S03: Tags are immutable write targets. + test { + sql """insert into branch_dml_boundary@branch(protected_tag) values (9, 'T', 'tag-write')""" + // TestAction stores only one `exception` substring, so one closure must + // verify both parts of the capability-boundary diagnostic. + check { result, exception, startTime, endTime -> + assertTrue(exception != null) + String message = exception.toString() + assertTrue(message.contains("tag")) + assertTrue(message.contains("not a branch")) + } + } + + // Cross-engine reads cover both the unchanged main ref and the branch + // after its successful writes and rejected row-level DML attempts. + spark_iceberg """refresh table demo.${dbName}.branch_dml_boundary""" + def sparkMainRows = spark_iceberg """ + select id, region, payload + from demo.${dbName}.branch_dml_boundary + order by id + """ + def dorisMainRows = sql """ + select id, region, payload + from branch_dml_boundary + order by id + """ + assertSparkDorisResultEquals(sparkMainRows, dorisMainRows) + def sparkBranchRows = spark_iceberg """ + select id, region, payload + from demo.${dbName}.branch_dml_boundary version as of 'audit_branch' + order by id + """ + def dorisBranchRows = sql """ + select id, region, payload + from branch_dml_boundary@branch(audit_branch) + order by id + """ + assertSparkDorisResultEquals(sparkBranchRows, dorisBranchRows) +} diff --git a/regression-test/suites/external_table_p0/iceberg/write/test_iceberg_write_complex_evolution.groovy b/regression-test/suites/external_table_p0/iceberg/write/test_iceberg_write_complex_evolution.groovy new file mode 100644 index 00000000000000..e5ad9e7c6ed19e --- /dev/null +++ b/regression-test/suites/external_table_p0/iceberg/write/test_iceberg_write_complex_evolution.groovy @@ -0,0 +1,178 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +suite("test_iceberg_write_complex_evolution", + "p0,external,iceberg,external_docker,external_docker_iceberg") { + String enabled = context.config.otherConfigs.get("enableIcebergTest") + if (enabled == null || !enabled.equalsIgnoreCase("true")) { + logger.info("disable iceberg test") + return + } + + String restPort = context.config.otherConfigs.get("iceberg_rest_uri_port") + String minioPort = context.config.otherConfigs.get("iceberg_minio_port") + String externalEnvIp = context.config.otherConfigs.get("externalEnvIp") + String catalogName = "test_iceberg_write_complex_evolution" + String dbName = "iceberg_write_complex_evolution_db" + + def assertSparkMatchesDoris = { + sql """refresh table ${dbName}.complex_evolution""" + spark_iceberg """refresh table demo.${dbName}.complex_evolution""" + def sparkRows = spark_iceberg """ + select id, group_key, arr, mp, payload + from demo.${dbName}.complex_evolution + order by id + """ + def dorisRows = sql """ + select id, group_key, arr, mp, payload + from complex_evolution + order by id + """ + assertSparkDorisResultEquals(sparkRows, dorisRows) + } + + sql """drop catalog if exists ${catalogName}""" + sql """ + create catalog ${catalogName} properties ( + "type" = "iceberg", + "iceberg.catalog.type" = "rest", + "uri" = "http://${externalEnvIp}:${restPort}", + "s3.access_key" = "admin", + "s3.secret_key" = "password", + "s3.endpoint" = "http://${externalEnvIp}:${minioPort}", + "s3.region" = "us-east-1", + "meta.cache.iceberg.table.ttl-second" = "0", + "meta.cache.iceberg.schema.ttl-second" = "0" + ) + """ + sql """switch ${catalogName}""" + sql """drop database if exists ${dbName} force""" + sql """create database ${dbName}""" + sql """use ${dbName}""" + sql """set enable_fallback_to_original_planner = false""" + + sql """drop table if exists complex_evolution""" + sql """ + create table complex_evolution ( + id int not null, + group_key string not null, + arr array, + mp map, + payload struct< + metric:int, + label:string, + nested:struct + > + ) + partition by list (group_key) () + properties ( + "format-version" = "2", + "write.format.default" = "orc" + ) + """ + + // W02-S01: Write old-schema rows, including NULL collections, elements, values and children. + sql """ + insert into complex_evolution values + (1, 'A', array(1, null, 3), map('x', 10, 'null-value', null), + struct(10, 'old-a', struct(1, null))), + (2, 'N', null, map('x', null), + struct(20, null, struct(null, 'old-null'))), + (3, 'B', array(), map(), null) + """ + String baseSnapshot = (sql """ + select snapshot_id from complex_evolution\$snapshots + order by committed_at desc limit 1 + """)[0][0].toString() + sql """alter table complex_evolution create tag complex_base as of version ${baseSnapshot}""" + assertSparkMatchesDoris() + + // W02-S02: Promote every supported nested primitive and add STRUCT children. + // The following write checks that Doris uses Iceberg field ids rather than child positions. + sql """alter table complex_evolution modify column arr array""" + sql """alter table complex_evolution modify column mp map""" + sql """ + alter table complex_evolution modify column payload struct< + metric:bigint, + label:string, + nested:struct, + tags:array, + attributes:map + > + """ + sql """alter table complex_evolution add partition key bucket(8, id) as id_bucket""" + sql """alter table complex_evolution add partition key truncate(1, group_key) as group_prefix""" + + sql """ + insert into complex_evolution values + (4, 'A1', array(cast(4000000000 as bigint), null), + map('large', cast(5000000000 as bigint), 'null-value', null), + struct( + cast(6000000000 as bigint), + 'new-a', + struct(cast(7000000000 as bigint), 'nested-new', cast(7.5 as double)), + array('x', null, 'z'), + map('a', cast(8000000000 as bigint), 'b', null) + )), + (5, 'N2', array(null), null, + struct( + cast(50 as bigint), + null, + struct(cast(5 as bigint), null, null), + null, + map('null-value', null) + )) + """ + + // W02-S03: Current schema reads both old and new files without moving old child values. + order_qt_complex_current """ + select id, group_key, arr, mp, payload + from complex_evolution + order by id + """ + order_qt_complex_children """ + select id, payload.metric, payload.nested.count, payload.nested.score, + payload.tags, payload.attributes + from complex_evolution + order by id + """ + order_qt_complex_nulls """ + select id + from complex_evolution + where group_key is null + or arr is null + or mp is null + or payload is null + or payload.nested.score is null + order by id + """ + order_qt_complex_partition_specs """ + select spec_id, sum(record_count) + from complex_evolution\$partitions + group by spec_id + order by spec_id + """ + assertSparkMatchesDoris() + + // W02-S04: A pre-evolution tag binds the old files to their historical complex schema. + order_qt_complex_base_tag """ + select id, arr, mp, payload.metric, payload.label, + payload.nested.count, payload.nested.comment + from complex_evolution@tag(complex_base) + order by id + """ +} diff --git a/regression-test/suites/external_table_p0/iceberg/write/test_iceberg_write_concurrent_merge_invariants.groovy b/regression-test/suites/external_table_p0/iceberg/write/test_iceberg_write_concurrent_merge_invariants.groovy new file mode 100644 index 00000000000000..6d46cf68a3db7d --- /dev/null +++ b/regression-test/suites/external_table_p0/iceberg/write/test_iceberg_write_concurrent_merge_invariants.groovy @@ -0,0 +1,209 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +import java.util.Collections +import java.util.concurrent.CountDownLatch +import java.util.concurrent.TimeUnit + +suite("test_iceberg_write_concurrent_merge_invariants", + "p0,external,iceberg,external_docker,external_docker_iceberg") { + String enabled = context.config.otherConfigs.get("enableIcebergTest") + if (enabled == null || !enabled.equalsIgnoreCase("true")) { + logger.info("disable iceberg test") + return + } + + String restPort = context.config.otherConfigs.get("iceberg_rest_uri_port") + String minioPort = context.config.otherConfigs.get("iceberg_minio_port") + String externalEnvIp = context.config.otherConfigs.get("externalEnvIp") + String catalogName = "test_iceberg_write_concurrent_merge_invariants" + String dbName = "iceberg_write_concurrent_merge_invariants_db" + + sql """drop catalog if exists ${catalogName}""" + sql """ + create catalog ${catalogName} properties ( + "type" = "iceberg", + "iceberg.catalog.type" = "rest", + "uri" = "http://${externalEnvIp}:${restPort}", + "s3.access_key" = "admin", + "s3.secret_key" = "password", + "s3.endpoint" = "http://${externalEnvIp}:${minioPort}", + "s3.region" = "us-east-1", + "meta.cache.iceberg.table.ttl-second" = "0", + "meta.cache.iceberg.schema.ttl-second" = "0" + ) + """ + sql """switch ${catalogName}""" + sql """drop database if exists ${dbName} force""" + sql """create database ${dbName}""" + sql """use ${dbName}""" + + sql """drop table if exists concurrent_merge""" + sql """ + create table concurrent_merge ( + id int not null, + region string, + payload string + ) + partition by list (region) () + properties ( + "format-version" = "2", + "write.format.default" = "parquet", + "write.delete.mode" = "merge-on-read", + "write.update.mode" = "merge-on-read", + "write.merge.mode" = "merge-on-read", + "write.merge.isolation-level" = "serializable" + ) + """ + sql """insert into concurrent_merge values (1, 'A', 'base')""" + long snapshotsBefore = (sql """select count(*) from concurrent_merge\$snapshots""")[0][0] as long + + def isExpectedIcebergCommitConflict = { Exception exception -> + String message = exception.toString() + return message.contains("org.apache.iceberg.exceptions.ValidationException") + || message.contains("org.apache.iceberg.exceptions.CommitFailedException") + || message.contains("Found conflicting files") + } + + // WC02-S01: A readiness barrier prevents a fast worker from completing + // before the second session is even eligible for concurrent dispatch. + // The exact winner is intentionally unspecified; cardinality, snapshot + // accounting and cross-engine visibility are deterministic invariants. + CountDownLatch ready = new CountDownLatch(2) + CountDownLatch start = new CountDownLatch(1) + List successes = Collections.synchronizedList(new ArrayList()) + List failures = Collections.synchronizedList(new ArrayList()) + + def first = thread("iceberg-merge-one") { + ready.countDown() + start.await() + try { + sql """ + merge into ${catalogName}.${dbName}.concurrent_merge t + using (select 1 as id, 'B' as region, 'winner-one' as payload) s + on t.id = s.id + when matched then update set region = s.region, payload = s.payload + """ + successes.add("one") + } catch (Exception e) { + // Only an Iceberg optimistic-validation conflict is an admissible + // loser; planner, RPC, catalog and BE failures must fail the suite. + if (!isExpectedIcebergCommitConflict(e)) { + throw e + } + failures.add(e.getMessage()) + } + } + def second = thread("iceberg-merge-two") { + ready.countDown() + start.await() + try { + sql """ + merge into ${catalogName}.${dbName}.concurrent_merge t + using (select 1 as id, 'C' as region, 'winner-two' as payload) s + on t.id = s.id + when matched then update set region = s.region, payload = s.payload + """ + successes.add("two") + } catch (Exception e) { + if (!isExpectedIcebergCommitConflict(e)) { + throw e + } + failures.add(e.getMessage()) + } + } + assertTrue(ready.await(30, TimeUnit.SECONDS), + "Both MERGE workers must reach the dispatch barrier") + start.countDown() + first.get() + second.get() + + assertTrue(successes.size() >= 1) + assertEquals(2, successes.size() + failures.size()) + assertEquals(1L, (sql """select count(*) from concurrent_merge where id = 1""")[0][0] as long) + assertEquals(snapshotsBefore + successes.size(), + (sql """select count(*) from concurrent_merge\$snapshots""")[0][0] as long) + def visible = sql """ + select payload + from concurrent_merge + where id = 1 + """ + assertTrue(["winner-one", "winner-two"].contains(visible[0][0].toString())) + + spark_iceberg """refresh table demo.${dbName}.concurrent_merge""" + def sparkRows = spark_iceberg """ + select id, region, payload + from demo.${dbName}.concurrent_merge + order by id + """ + def dorisRows = sql """ + select id, region, payload + from concurrent_merge + order by id + """ + assertSparkDorisResultEquals(sparkRows, dorisRows) + + // WC02-S02: Use the same readiness invariant for non-conflicting appends; + // otherwise sequential execution can falsely satisfy the row-count oracle. + CountDownLatch appendReady = new CountDownLatch(2) + CountDownLatch appendStart = new CountDownLatch(1) + def appendOne = thread("iceberg-append-one") { + appendReady.countDown() + appendStart.await() + sql """ + insert into ${catalogName}.${dbName}.concurrent_merge + select number + 10, 'append-one', concat('one-', number) + from numbers('number' = '128') + """ + } + def appendTwo = thread("iceberg-append-two") { + appendReady.countDown() + appendStart.await() + sql """ + insert into ${catalogName}.${dbName}.concurrent_merge + select number + 1000, 'append-two', concat('two-', number) + from numbers('number' = '128') + """ + } + assertTrue(appendReady.await(30, TimeUnit.SECONDS), + "Both append workers must reach the dispatch barrier") + appendStart.countDown() + appendOne.get() + appendTwo.get() + order_qt_concurrent_append_counts """ + select region, count(*), count(distinct id) + from concurrent_merge + where region in ('append-one', 'append-two') + group by region + order by region + """ + + // Refresh after both appends so the cross-engine oracle covers the + // concurrent commits rather than only the earlier MERGE result. + spark_iceberg """refresh table demo.${dbName}.concurrent_merge""" + sparkRows = spark_iceberg """ + select id, region, payload + from demo.${dbName}.concurrent_merge + order by id + """ + dorisRows = sql """ + select id, region, payload + from concurrent_merge + order by id + """ + assertSparkDorisResultEquals(sparkRows, dorisRows) +} diff --git a/regression-test/suites/external_table_p0/iceberg/write/test_iceberg_write_ctas_format_boundary.groovy b/regression-test/suites/external_table_p0/iceberg/write/test_iceberg_write_ctas_format_boundary.groovy new file mode 100644 index 00000000000000..9be6f3ba421281 --- /dev/null +++ b/regression-test/suites/external_table_p0/iceberg/write/test_iceberg_write_ctas_format_boundary.groovy @@ -0,0 +1,180 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +suite("test_iceberg_write_ctas_format_boundary", + "p0,external,iceberg,external_docker,external_docker_iceberg") { + String enabled = context.config.otherConfigs.get("enableIcebergTest") + if (enabled == null || !enabled.equalsIgnoreCase("true")) { + logger.info("disable iceberg test") + return + } + + String restPort = context.config.otherConfigs.get("iceberg_rest_uri_port") + String minioPort = context.config.otherConfigs.get("iceberg_minio_port") + String externalEnvIp = context.config.otherConfigs.get("externalEnvIp") + String catalogName = "test_iceberg_write_ctas_format_boundary" + String dbName = "iceberg_write_ctas_format_boundary_db" + String internalDbName = "iceberg_write_ctas_format_boundary_internal_db" + + sql """drop catalog if exists ${catalogName}""" + sql """ + create catalog ${catalogName} properties ( + "type" = "iceberg", + "iceberg.catalog.type" = "rest", + "uri" = "http://${externalEnvIp}:${restPort}", + "s3.access_key" = "admin", + "s3.secret_key" = "password", + "s3.endpoint" = "http://${externalEnvIp}:${minioPort}", + "s3.region" = "us-east-1", + "meta.cache.iceberg.table.ttl-second" = "0", + "meta.cache.iceberg.schema.ttl-second" = "0" + ) + """ + + sql """drop database if exists internal.${internalDbName} force""" + sql """create database internal.${internalDbName}""" + sql """drop table if exists internal.${internalDbName}.ctas_source""" + sql """ + create table internal.${internalDbName}.ctas_source ( + id int, + region varchar(20), + tags array, + attrs map, + detail struct + ) + duplicate key(id) + distributed by hash(id) buckets 4 + properties ("replication_num" = "1") + """ + sql """ + insert into internal.${internalDbName}.ctas_source values + (1, 'A', ['x', null], map('k', 'v'), struct(10, 'one')), + (2, null, [], map('null-value', null), struct(null, 'two')), + (3, '中文', ['😀'], map(), struct(30, null)) + """ + + sql """switch ${catalogName}""" + sql """drop database if exists ${dbName} force""" + sql """create database ${dbName}""" + sql """use ${dbName}""" + + // WC01-S01: CTAS preserves complex types, NULL values, partitioning and + // writer properties when the source is a distributed Doris table. + sql """drop table if exists ctas_complex_partitioned""" + sql """ + create table ctas_complex_partitioned + partition by list (region, bucket(4, id)) () + properties ( + "format-version" = "2", + "write.format.default" = "orc", + "write.orc.compression-codec" = "lz4" + ) + as + select id, + cast(if(id in (1, 3), 'A', region) as string) as region, + tags, attrs, detail + from internal.${internalDbName}.ctas_source + """ + order_qt_ctas_complex_rows """ + select id, region, tags, attrs, detail + from ctas_complex_partitioned + order by id + """ + order_qt_ctas_complex_files """ + select lower(file_format), sum(record_count) + from ctas_complex_partitioned\$files + group by lower(file_format) + order by lower(file_format) + """ + order_qt_ctas_complex_partitions """ + select spec_id, count(*), sum(record_count) + from ctas_complex_partitioned\$partitions + group by spec_id + order by spec_id + """ + order_qt_ctas_complex_physical_partitions """ + select struct_element(`partition`, 'region'), + struct_element(`partition`, 'id_bucket'), + record_count + from ctas_complex_partitioned\$partitions + order by 1, 2 + """ + // Two rows share identity region A, so more than one physical partition + // proves bucket(4,id) contributes to CTAS routing. + assertTrue(((sql """ + select count(distinct struct_element(`partition`, 'id_bucket')) + from ctas_complex_partitioned\$partitions + where struct_element(`partition`, 'region') = 'A' + """)[0][0] as long) > 1L) + spark_iceberg """refresh table demo.${dbName}.ctas_complex_partitioned""" + def sparkRows = spark_iceberg """ + select id, region, tags, attrs, detail + from demo.${dbName}.ctas_complex_partitioned + order by id + """ + def dorisRows = sql """ + select id, region, tags, attrs, detail + from ctas_complex_partitioned + order by id + """ + assertSparkDorisResultEquals(sparkRows, dorisRows) + def compressionProperty = spark_iceberg """ + show tblproperties demo.${dbName}.ctas_complex_partitioned + ('write.orc.compression-codec') + """ + // File-format checks do not prove the requested codec survived CTAS. + assertEquals("lz4", compressionProperty[0][1].toString().toLowerCase()) + + // WC01-S02: CTAS is atomic. A source expression failure must not leave a + // visible Iceberg table or a partially committed snapshot. + sql """set enable_strict_cast = true""" + sql """drop table if exists ctas_failed_atomicity""" + test { + sql """ + create table ctas_failed_atomicity + properties ("format-version" = "2") + as + select cast(if(number = 2, 'invalid-id', cast(number as string)) as int) as id, + concat('candidate-', number) as payload + from numbers('number' = '8') + """ + exception "can't cast to INT in strict mode" + } + assertEquals(0, (sql """show tables like 'ctas_failed_atomicity'""").size()) + + // WC01-S03: Iceberg allows Avro, but the current Doris writer supports + // Parquet and ORC only. Reject Avro explicitly instead of silently falling back. + sql """drop table if exists avro_write_boundary""" + sql """ + create table avro_write_boundary ( + id int, + payload string + ) + properties ( + "format-version" = "2", + "write.format.default" = "avro" + ) + """ + long avroSnapshots = (sql """select count(*) from avro_write_boundary\$snapshots""")[0][0] as long + test { + sql """insert into avro_write_boundary values (1, 'must-not-fallback')""" + exception "Unsupported input format type: avro" + } + assertEquals(avroSnapshots, + (sql """select count(*) from avro_write_boundary\$snapshots""")[0][0] as long) + assertEquals(0, (sql """select count(*) from avro_write_boundary\$files""")[0][0] as long) +} diff --git a/regression-test/suites/external_table_p0/iceberg/write/test_iceberg_write_dml_modes_evolution.groovy b/regression-test/suites/external_table_p0/iceberg/write/test_iceberg_write_dml_modes_evolution.groovy new file mode 100644 index 00000000000000..db999e19651e5c --- /dev/null +++ b/regression-test/suites/external_table_p0/iceberg/write/test_iceberg_write_dml_modes_evolution.groovy @@ -0,0 +1,269 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +suite("test_iceberg_write_dml_modes_evolution", + "p0,external,iceberg,external_docker,external_docker_iceberg") { + String enabled = context.config.otherConfigs.get("enableIcebergTest") + if (enabled == null || !enabled.equalsIgnoreCase("true")) { + logger.info("disable iceberg test") + return + } + + String restPort = context.config.otherConfigs.get("iceberg_rest_uri_port") + String minioPort = context.config.otherConfigs.get("iceberg_minio_port") + String externalEnvIp = context.config.otherConfigs.get("externalEnvIp") + String catalogName = "test_iceberg_write_dml_modes_evolution" + String dbName = "iceberg_write_dml_modes_evolution_db" + + def assertSparkMatchesDoris = { String tableName -> + sql """refresh table ${dbName}.${tableName}""" + spark_iceberg """refresh table demo.${dbName}.${tableName}""" + def sparkRows = spark_iceberg """ + select id, region, bucket_key, event_time, score, status + from demo.${dbName}.${tableName} + order by id + """ + def dorisRows = sql """ + select id, region, bucket_key, event_time, score, status + from ${tableName} + order by id + """ + assertSparkDorisResultEquals(sparkRows, dorisRows) + } + + sql """drop catalog if exists ${catalogName}""" + sql """ + create catalog ${catalogName} properties ( + "type" = "iceberg", + "iceberg.catalog.type" = "rest", + "uri" = "http://${externalEnvIp}:${restPort}", + "s3.access_key" = "admin", + "s3.secret_key" = "password", + "s3.endpoint" = "http://${externalEnvIp}:${minioPort}", + "s3.region" = "us-east-1", + "meta.cache.iceberg.table.ttl-second" = "0", + "meta.cache.iceberg.schema.ttl-second" = "0" + ) + """ + sql """switch ${catalogName}""" + sql """drop database if exists ${dbName} force""" + sql """create database ${dbName}""" + sql """use ${dbName}""" + sql """set enable_fallback_to_original_planner = false""" + + sql """drop table if exists mor_evolution""" + sql """ + create table mor_evolution ( + id int not null, + region string, + bucket_key string not null, + event_time datetime, + score int + ) + partition by list (region, bucket(4, bucket_key), day(event_time)) () + properties ( + "format-version" = "2", + "write.format.default" = "parquet", + "write.delete.mode" = "merge-on-read", + "write.update.mode" = "merge-on-read", + "write.merge.mode" = "merge-on-read" + ) + """ + + // W03-S01: The MOR baseline includes NULL in every partition transform family. + sql """ + insert into mor_evolution values + (1, 'A', 'alpha', '2026-01-01 01:00:00', 10), + (2, 'B', 'beta', '2026-01-02 02:00:00', 20), + (3, null, 'null-key', null, 30), + (4, 'A', 'delta', '2026-01-04 04:00:00', 40) + """ + String morBaseSnapshot = (sql """ + select snapshot_id from mor_evolution\$snapshots + order by committed_at desc limit 1 + """)[0][0].toString() + sql """alter table mor_evolution create tag mor_base as of version ${morBaseSnapshot}""" + + // W03-S02: Evolve schema and partition spec, then write more files before row-level DML. + sql """alter table mor_evolution add column status string""" + sql """ + alter table mor_evolution + replace partition key day(event_time) with month(event_time) as event_month + """ + sql """ + alter table mor_evolution + replace partition key bucket(4, bucket_key) with bucket(8, id) as id_bucket + """ + sql """ + insert into mor_evolution values + (5, 'B', 'echo', '2026-02-01 05:00:00', 50, 'new-spec'), + (6, null, 'foxtrot', null, 60, 'new-null'), + (7, 'C', 'golf', '2026-03-01 07:00:00', 70, null) + """ + String morBeforeDmlSnapshot = (sql """ + select snapshot_id from mor_evolution\$snapshots + order by committed_at desc limit 1 + """)[0][0].toString() + sql """alter table mor_evolution create tag mor_before_dml as of version ${morBeforeDmlSnapshot}""" + + // W03-S03: DELETE spans old/new specs and removes NULL partition rows. + sql """delete from mor_evolution where region is null""" + + // W03-S04: UPDATE changes partition source values in files from both specs. + sql """ + update mor_evolution + set region = concat(region, '-updated'), + score = score + 100, + status = 'updated' + where region = 'A' + """ + + // W03-S05: MERGE deletes, updates and inserts across different transformed partitions. + sql """ + merge into mor_evolution t + using ( + select 2 as id, 'B-merged' as region, 'beta-merged' as bucket_key, + timestamp '2026-04-02 02:00:00' as event_time, 220 as score, + 'U' as op + union all + select 5, 'B', 'echo', timestamp '2026-02-01 05:00:00', 50, 'D' + union all + select 8, 'D', 'hotel', timestamp '2026-05-01 08:00:00', 80, 'I' + ) s + on t.id = s.id + when matched and s.op = 'D' then delete + when matched then update set + region = s.region, + bucket_key = s.bucket_key, + event_time = s.event_time, + score = s.score, + status = 'merged' + when not matched then insert (id, region, bucket_key, event_time, score, status) + values (s.id, s.region, s.bucket_key, s.event_time, s.score, 'inserted') + """ + + order_qt_mor_current """ + select id, region, bucket_key, event_time, score, status + from mor_evolution + order by id + """ + order_qt_mor_base_tag """ + select id, region, bucket_key, event_time, score + from mor_evolution@tag(mor_base) + order by id + """ + order_qt_mor_before_dml_tag """ + select id, region, bucket_key, event_time, score, status + from mor_evolution@tag(mor_before_dml) + order by id + """ + order_qt_mor_delete_files """ + select spec_id, count(*), sum(record_count) + from mor_evolution\$delete_files + group by spec_id + order by spec_id + """ + assertSparkMatchesDoris("mor_evolution") + + // W03-S06: COW accepts INSERT after partition evolution but Doris explicitly rejects + // DELETE/UPDATE/MERGE. Each rejection must leave both data and snapshot count unchanged. + sql """drop table if exists cow_evolution""" + sql """ + create table cow_evolution ( + id int not null, + region string, + bucket_key string not null, + event_time datetime, + score int, + status string + ) + partition by list (region, bucket(4, bucket_key), day(event_time)) () + properties ( + "format-version" = "2", + "write.format.default" = "orc", + "write.delete.mode" = "copy-on-write", + "write.update.mode" = "copy-on-write", + "write.merge.mode" = "copy-on-write" + ) + """ + sql """ + insert into cow_evolution values + (1, 'A', 'alpha', '2026-01-01 01:00:00', 10, 'base'), + (2, null, 'null-key', null, 20, 'null-partition') + """ + sql """ + alter table cow_evolution + replace partition key bucket(4, bucket_key) with bucket(8, id) as id_bucket + """ + sql """ + alter table cow_evolution + replace partition key day(event_time) with month(event_time) as event_month + """ + sql """ + insert into cow_evolution values + (3, 'B', 'beta', '2026-02-01 03:00:00', 30, 'new-spec') + """ + + long cowSnapshots = (sql """select count(*) from cow_evolution\$snapshots""")[0][0] as long + test { + sql """delete from cow_evolution where region is null""" + // TestAction has one exception slot; a check closure preserves both + // the rejected operation and its actionable property guidance. + check { result, exception, startTime, endTime -> + assertTrue(exception != null) + String message = exception.toString() + assertTrue(message.contains( + "Doris does not support DELETE on Iceberg copy-on-write tables")) + assertTrue(message.contains( + "Set table property 'write.delete.mode' to 'merge-on-read'")) + } + } + test { + sql """update cow_evolution set score = score + 1 where id = 1""" + check { result, exception, startTime, endTime -> + assertTrue(exception != null) + String message = exception.toString() + assertTrue(message.contains( + "Doris does not support UPDATE on Iceberg copy-on-write tables")) + assertTrue(message.contains( + "Set table property 'write.update.mode' to 'merge-on-read'")) + } + } + test { + sql """ + merge into cow_evolution t + using (select 1 as id, 100 as score) s + on t.id = s.id + when matched then update set score = s.score + """ + check { result, exception, startTime, endTime -> + assertTrue(exception != null) + String message = exception.toString() + assertTrue(message.contains( + "Doris does not support MERGE INTO on Iceberg copy-on-write tables")) + assertTrue(message.contains( + "Set table property 'write.merge.mode' to 'merge-on-read'")) + } + } + assertEquals(cowSnapshots, (sql """select count(*) from cow_evolution\$snapshots""")[0][0] as long) + order_qt_cow_after_rejections """ + select id, region, bucket_key, event_time, score, status + from cow_evolution + order by id + """ + assertSparkMatchesDoris("cow_evolution") +} diff --git a/regression-test/suites/external_table_p0/iceberg/write/test_iceberg_write_evolution_refs.groovy b/regression-test/suites/external_table_p0/iceberg/write/test_iceberg_write_evolution_refs.groovy new file mode 100644 index 00000000000000..3d6ec5a2616335 --- /dev/null +++ b/regression-test/suites/external_table_p0/iceberg/write/test_iceberg_write_evolution_refs.groovy @@ -0,0 +1,253 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +suite("test_iceberg_write_evolution_refs", + "p0,external,iceberg,external_docker,external_docker_iceberg") { + String enabled = context.config.otherConfigs.get("enableIcebergTest") + if (enabled == null || !enabled.equalsIgnoreCase("true")) { + logger.info("disable iceberg test") + return + } + + String restPort = context.config.otherConfigs.get("iceberg_rest_uri_port") + String minioPort = context.config.otherConfigs.get("iceberg_minio_port") + String externalEnvIp = context.config.otherConfigs.get("externalEnvIp") + String catalogName = "test_iceberg_write_evolution_refs" + String dbName = "iceberg_write_evolution_refs_db" + + def latestSnapshotId = { + return (sql """ + select snapshot_id + from evolution_refs\$snapshots + order by committed_at desc + limit 1 + """)[0][0].toString() + } + + def assertSparkMatchesDoris = { String relation, String projection -> + sql """refresh table ${dbName}.evolution_refs""" + spark_iceberg """refresh table demo.${dbName}.evolution_refs""" + def sparkRows = spark_iceberg """ + select ${projection} + from demo.${dbName}.evolution_refs${relation} + order by id + """ + def dorisRows = sql """ + select ${projection} + from evolution_refs${relation} + order by id + """ + assertSparkDorisResultEquals(sparkRows, dorisRows) + } + + def assertSparkBranchMatchesDoris = { String branch, String projection -> + sql """refresh table ${dbName}.evolution_refs""" + spark_iceberg """refresh table demo.${dbName}.evolution_refs""" + def sparkRows = spark_iceberg """ + select ${projection} + from demo.${dbName}.evolution_refs version as of '${branch}' + order by id + """ + def dorisRows = sql """ + select ${projection} + from evolution_refs@branch(${branch}) + order by id + """ + assertSparkDorisResultEquals(sparkRows, dorisRows) + } + + sql """drop catalog if exists ${catalogName}""" + sql """ + create catalog ${catalogName} properties ( + "type" = "iceberg", + "iceberg.catalog.type" = "rest", + "uri" = "http://${externalEnvIp}:${restPort}", + "s3.access_key" = "admin", + "s3.secret_key" = "password", + "s3.endpoint" = "http://${externalEnvIp}:${minioPort}", + "s3.region" = "us-east-1", + "meta.cache.iceberg.table.ttl-second" = "0", + "meta.cache.iceberg.schema.ttl-second" = "0" + ) + """ + sql """switch ${catalogName}""" + sql """drop database if exists ${dbName} force""" + sql """create database ${dbName}""" + sql """use ${dbName}""" + sql """set enable_fallback_to_original_planner = false""" + + sql """drop table if exists evolution_refs""" + sql """ + create table evolution_refs ( + id int not null, + region string, + bucket_key string not null, + event_time datetime, + amount decimal(12, 2), + payload struct + ) + partition by list (region, bucket(4, bucket_key), day(event_time)) () + properties ( + "format-version" = "2", + "write.format.default" = "parquet", + "write.delete.mode" = "merge-on-read", + "write.update.mode" = "merge-on-read", + "write.merge.mode" = "merge-on-read" + ) + """ + + // W01-S01: Doris writes the first snapshot using identity, string bucket and day transforms. + sql """ + insert into evolution_refs values + (1, 'CN', 'alpha', '2026-01-01 08:00:00', 10.10, struct(10, 'base-cn')), + (2, 'US', 'beta', '2026-01-02 09:00:00', 20.20, struct(20, 'base-us')), + (3, null, 'null-key', null, 30.30, struct(30, null)) + """ + String baseSnapshot = latestSnapshotId() + sql """alter table evolution_refs create tag base_tag as of version ${baseSnapshot}""" + sql """alter table evolution_refs create branch base_branch as of version ${baseSnapshot}""" + assertSparkMatchesDoris("", "id, region, bucket_key, event_time, amount") + + // W01-S02: Schema and partition spec evolve together before the next Doris write. + // Renaming the partition source column must preserve its Iceberg field id. + sql """alter table evolution_refs add column note string""" + sql """alter table evolution_refs rename column region zone""" + sql """ + alter table evolution_refs modify column payload struct< + metric:bigint, + label:string, + extra:string + > + """ + sql """ + alter table evolution_refs + replace partition key day(event_time) with month(event_time) as event_month + """ + sql """ + alter table evolution_refs + replace partition key bucket(4, bucket_key) with bucket(8, id) as id_bucket + """ + sql """alter table evolution_refs drop partition key region""" + sql """alter table evolution_refs add partition key truncate(2, bucket_key) as bucket_prefix""" + + sql """ + insert into evolution_refs values + (4, 'CN-east', 'gamma', '2026-02-01 10:00:00', 40.40, + struct(4000000000, 'new-cn', 'after-evolution'), 'new-spec'), + (5, 'DE-west', 'delta', '2026-03-02 11:00:00', 50.50, + struct(50, 'new-de', null), null), + (6, null, 'epsilon', null, 60.60, + struct(60, null, 'null-partition'), 'null-zone') + """ + String evolvedSnapshot = latestSnapshotId() + sql """alter table evolution_refs create tag evolved_tag as of version ${evolvedSnapshot}""" + + // W01-S03: Source-column filters must cover files written with both partition specs. + order_qt_current_rows """ + select id, zone, bucket_key, event_time, amount, payload.metric, payload.extra, note + from evolution_refs + order by id + """ + order_qt_cross_spec_zone_filter """ + select id from evolution_refs + where zone = 'CN' or zone like 'CN-%' + order by id + """ + order_qt_cross_spec_time_filter """ + select id from evolution_refs + where event_time is null or event_time >= timestamp '2026-02-01 00:00:00' + order by id + """ + order_qt_partition_specs """ + select spec_id, sum(record_count) + from evolution_refs\$partitions + group by spec_id + order by spec_id + """ + assertSparkMatchesDoris("", "id, zone, bucket_key, event_time, amount") + + // W01-S04: Numeric snapshot and tag retain both base data and the historical schema. + order_qt_base_snapshot """ + select id, region + from evolution_refs for version as of ${baseSnapshot} + order by id + """ + order_qt_base_tag """ + select id, region + from evolution_refs@tag(base_tag) + order by id + """ + order_qt_evolved_tag """ + select id, zone, note + from evolution_refs@tag(evolved_tag) + order by id + """ + + // W01-S05: Seed the current-spec branch partition before overwriting it; + // this proves replacement semantics while main and the base tag stay isolated. + sql """ + insert into evolution_refs@branch(base_branch) + (id, zone, bucket_key, event_time, amount, payload, note) + values + (7, 'JP-east', 'branch-a', '2026-04-01 12:00:00', 70.70, + struct(70, 'branch', 'current-schema'), 'branch-insert'), + (8, 'FR-west', 'branch-b', '2026-05-01 13:00:00', 80.80, + struct(80, 'branch-seed', 'current-schema'), 'branch-overwrite-seed') + """ + order_qt_branch_after_insert """ + select id, zone, note + from evolution_refs@branch(base_branch) + order by id + """ + order_qt_main_unchanged_after_branch_insert """ + select id from evolution_refs order by id + """ + assertSparkBranchMatchesDoris( + "base_branch", + "id, zone, bucket_key, event_time, amount, note") + + sql """ + insert overwrite table evolution_refs@branch(base_branch) + select 8, 'FR-west', 'branch-b', timestamp '2026-05-01 13:00:00', + cast(80.80 as decimal(12, 2)), + struct(cast(80 as bigint), 'branch-overwrite', 'current-schema'), + 'branch-overwrite' + """ + assertEquals(0L, (sql """ + select count(*) from evolution_refs@branch(base_branch) + where note = 'branch-overwrite-seed' + """)[0][0] as long) + order_qt_branch_after_overwrite """ + select id, zone, note + from evolution_refs@branch(base_branch) + order by id + """ + order_qt_base_tag_after_branch_overwrite """ + select id, region + from evolution_refs@tag(base_tag) + order by id + """ + order_qt_main_after_branch_overwrite """ + select id, zone, note + from evolution_refs + order by id + """ + assertSparkBranchMatchesDoris( + "base_branch", + "id, zone, bucket_key, event_time, amount, note") + assertSparkMatchesDoris("", "id, zone, bucket_key, event_time, amount") +} diff --git a/regression-test/suites/external_table_p0/iceberg/write/test_iceberg_write_merge_duplicate_source_negative.groovy b/regression-test/suites/external_table_p0/iceberg/write/test_iceberg_write_merge_duplicate_source_negative.groovy new file mode 100644 index 00000000000000..fbb3be22c44834 --- /dev/null +++ b/regression-test/suites/external_table_p0/iceberg/write/test_iceberg_write_merge_duplicate_source_negative.groovy @@ -0,0 +1,102 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +suite("test_iceberg_write_merge_duplicate_source_negative", + "p0,external,iceberg,external_docker,external_docker_iceberg") { + String enabled = context.config.otherConfigs.get("enableIcebergTest") + if (enabled == null || !enabled.equalsIgnoreCase("true")) { + logger.info("disable iceberg test") + return + } + String knownBugEnabled = context.config.otherConfigs.get("enableIcebergKnownBugTest") + if (knownBugEnabled == null || !knownBugEnabled.equalsIgnoreCase("true")) { + logger.info("skip isolated Iceberg known-bug test") + return + } + + String restPort = context.config.otherConfigs.get("iceberg_rest_uri_port") + String minioPort = context.config.otherConfigs.get("iceberg_minio_port") + String externalEnvIp = context.config.otherConfigs.get("externalEnvIp") + String catalogName = "test_iceberg_write_merge_duplicate_source_negative" + String dbName = "iceberg_write_merge_duplicate_source_negative_db" + + sql """drop catalog if exists ${catalogName}""" + sql """ + create catalog ${catalogName} properties ( + "type" = "iceberg", + "iceberg.catalog.type" = "rest", + "uri" = "http://${externalEnvIp}:${restPort}", + "s3.access_key" = "admin", + "s3.secret_key" = "password", + "s3.endpoint" = "http://${externalEnvIp}:${minioPort}", + "s3.region" = "us-east-1" + ) + """ + sql """switch ${catalogName}""" + sql """drop database if exists ${dbName} force""" + sql """create database ${dbName}""" + sql """use ${dbName}""" + + sql """drop table if exists duplicate_source_target""" + sql """ + create table duplicate_source_target ( + id int, + region string, + payload string + ) + partition by list (region) () + properties ( + "format-version" = "2", + "write.delete.mode" = "merge-on-read", + "write.update.mode" = "merge-on-read", + "write.merge.mode" = "merge-on-read" + ) + """ + sql """insert into duplicate_source_target values (1, 'A', 'committed')""" + + long snapshotsBefore = + (sql """select count(*) from duplicate_source_target\$snapshots""")[0][0] as long + long filesBefore = + (sql """select count(*) from duplicate_source_target\$files""")[0][0] as long + + // Negative scenario: Iceberg MERGE cardinality permits only one source row + // to update a target row. The entire statement must fail before publishing. + test { + sql """ + merge into duplicate_source_target t + using ( + select 1 as id, 'B' as region, 'first-update' as payload + union all + select 1, 'C', 'second-update' + ) s + on t.id = s.id + when matched then update set + region = s.region, + payload = s.payload + """ + exception "more than one" + } + assertEquals(snapshotsBefore, + (sql """select count(*) from duplicate_source_target\$snapshots""")[0][0] as long) + assertEquals(filesBefore, + (sql """select count(*) from duplicate_source_target\$files""")[0][0] as long) + order_qt_duplicate_source_atomic_state """ + select id, region, payload + from duplicate_source_target + order by id + """ +} diff --git a/regression-test/suites/external_table_p0/iceberg/write/test_iceberg_write_merge_semantics.groovy b/regression-test/suites/external_table_p0/iceberg/write/test_iceberg_write_merge_semantics.groovy new file mode 100644 index 00000000000000..0bdac94f8959ea --- /dev/null +++ b/regression-test/suites/external_table_p0/iceberg/write/test_iceberg_write_merge_semantics.groovy @@ -0,0 +1,207 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +suite("test_iceberg_write_merge_semantics", + "p0,external,iceberg,external_docker,external_docker_iceberg") { + String enabled = context.config.otherConfigs.get("enableIcebergTest") + if (enabled == null || !enabled.equalsIgnoreCase("true")) { + logger.info("disable iceberg test") + return + } + + String restPort = context.config.otherConfigs.get("iceberg_rest_uri_port") + String minioPort = context.config.otherConfigs.get("iceberg_minio_port") + String externalEnvIp = context.config.otherConfigs.get("externalEnvIp") + String catalogName = "test_iceberg_write_merge_semantics" + String dbName = "iceberg_write_merge_semantics_db" + + def assertSparkMatchesDoris = { String tableName -> + sql """refresh table ${dbName}.${tableName}""" + spark_iceberg """refresh table demo.${dbName}.${tableName}""" + def sparkRows = spark_iceberg """ + select id, p_identity, p_bucket, p_truncate, payload, status + from demo.${dbName}.${tableName} + order by id, payload + """ + def dorisRows = sql """ + select id, p_identity, p_bucket, p_truncate, payload, status + from ${tableName} + order by id, payload + """ + assertSparkDorisResultEquals(sparkRows, dorisRows) + } + + sql """drop catalog if exists ${catalogName}""" + sql """ + create catalog ${catalogName} properties ( + "type" = "iceberg", + "iceberg.catalog.type" = "rest", + "uri" = "http://${externalEnvIp}:${restPort}", + "s3.access_key" = "admin", + "s3.secret_key" = "password", + "s3.endpoint" = "http://${externalEnvIp}:${minioPort}", + "s3.region" = "us-east-1", + "meta.cache.iceberg.table.ttl-second" = "0", + "meta.cache.iceberg.schema.ttl-second" = "0" + ) + """ + sql """switch ${catalogName}""" + sql """drop database if exists ${dbName} force""" + sql """create database ${dbName}""" + sql """use ${dbName}""" + + sql """drop table if exists merge_semantics""" + sql """ + create table merge_semantics ( + id int, + p_identity string, + p_bucket string, + p_truncate string not null, + payload string, + status string + ) + partition by list ( + p_identity, + bucket(8, p_bucket) + ) () + properties ( + "format-version" = "2", + "write.format.default" = "parquet", + "write.delete.mode" = "merge-on-read", + "write.update.mode" = "merge-on-read", + "write.merge.mode" = "merge-on-read" + ) + """ + sql """ + insert into merge_semantics values + (1, 'A', 'bucket-a', 'alpha', 'old-1', 'active'), + (2, null, 'bucket-b', 'beta', 'old-2', 'active'), + (3, 'C', 'bucket-c', 'charlie', 'old-3', 'active') + """ + + // WM01-S01: Conditions on MATCHED and NOT MATCHED clauses select exactly one + // action, including an update that moves a row across STRING identity and bucket transforms. + sql """ + merge into merge_semantics t + using ( + select 1 as id, 'A2' as p_identity, 'bucket-a2' as p_bucket, + 'delta' as p_truncate, 'new-1' as payload, 'U' as op, true as accepted + union all + select 3, 'C', 'bucket-c', 'charlie', 'old-3', 'D', true + union all + select 4, null, 'bucket-d', 'echo', 'new-4', 'I1', true + union all + select 5, 'E', 'bucket-e', 'foxtrot', 'new-5', 'I2', true + union all + select 6, 'F', 'bucket-f', 'golf', 'filtered-6', 'I1', false + ) s + on t.id = s.id + when matched and s.op = 'D' then delete + when matched and s.op = 'U' then update set + p_identity = s.p_identity, + p_bucket = s.p_bucket, + p_truncate = s.p_truncate, + payload = s.payload, + status = 'updated' + when not matched and s.op = 'I1' and s.accepted then + insert (id, p_identity, p_bucket, p_truncate, payload, status) + values (s.id, s.p_identity, s.p_bucket, s.p_truncate, s.payload, 'insert-1') + when not matched and s.op = 'I2' and s.accepted then + insert (id, p_identity, p_bucket, p_truncate, payload, status) + values (s.id, s.p_identity, s.p_bucket, s.p_truncate, s.payload, 'insert-2') + """ + order_qt_merge_conditional_clauses """ + select id, p_identity, p_bucket, p_truncate, payload, status + from merge_semantics + order by id + """ + order_qt_merge_string_partition_metadata """ + select spec_id, count(*), sum(record_count) + from merge_semantics\$partitions + group by spec_id + order by spec_id + """ + assertSparkMatchesDoris("merge_semantics") + + // WM01-S02: NULL-safe equality updates one nullable key while ordinary + // equality leaves NULL unmatched and executes the NOT MATCHED action. + sql """drop table if exists merge_null_keys""" + sql """ + create table merge_null_keys ( + id int, + p_identity string, + p_bucket string, + p_truncate string, + payload string, + status string + ) + properties ( + "format-version" = "2", + "write.format.default" = "orc", + "write.delete.mode" = "merge-on-read", + "write.update.mode" = "merge-on-read", + "write.merge.mode" = "merge-on-read" + ) + """ + sql """insert into merge_null_keys values (null, null, null, null, 'target-null', 'old')""" + sql """ + merge into merge_null_keys t + using ( + select cast(null as int) as id, cast(null as string) as p_identity, + cast(null as string) as p_bucket, cast(null as string) as p_truncate, + 'source-null-safe' as payload + ) s + on t.id <=> s.id + when matched then update set payload = s.payload, status = 'null-safe-update' + """ + sql """ + merge into merge_null_keys t + using ( + select cast(null as int) as id, cast(null as string) as p_identity, + cast(null as string) as p_bucket, cast(null as string) as p_truncate, + 'source-ordinary' as payload + ) s + on t.id = s.id + when matched then update set payload = 'must-not-update' + when not matched then + insert (id, p_identity, p_bucket, p_truncate, payload, status) + values (s.id, s.p_identity, s.p_bucket, s.p_truncate, s.payload, 'ordinary-insert') + """ + order_qt_merge_null_keys """ + select id, p_identity, p_bucket, p_truncate, payload, status + from merge_null_keys + order by payload + """ + assertSparkMatchesDoris("merge_null_keys") + + // WM01-S03: An unconditional clause must be last within its clause family; + // otherwise a later conditional clause is unreachable. + long snapshotsBeforeInvalidClause = + (sql """select count(*) from merge_semantics\$snapshots""")[0][0] as long + test { + sql """ + merge into merge_semantics t + using (select 2 as id, 'X' as payload) s + on t.id = s.id + when matched then update set payload = s.payload + when matched and s.payload = 'X' then delete + """ + exception "Only the last matched clause could without case predicate" + } + assertEquals(snapshotsBeforeInvalidClause, + (sql """select count(*) from merge_semantics\$snapshots""")[0][0] as long) +} diff --git a/regression-test/suites/external_table_p0/iceberg/write/test_iceberg_write_merge_truncate_negative.groovy b/regression-test/suites/external_table_p0/iceberg/write/test_iceberg_write_merge_truncate_negative.groovy new file mode 100644 index 00000000000000..31bfeb251ab609 --- /dev/null +++ b/regression-test/suites/external_table_p0/iceberg/write/test_iceberg_write_merge_truncate_negative.groovy @@ -0,0 +1,100 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +suite("test_iceberg_write_merge_truncate_negative", + "p0,external,iceberg,external_docker,external_docker_iceberg") { + String enabled = context.config.otherConfigs.get("enableIcebergTest") + String crashTestEnabled = context.config.otherConfigs.get("enableIcebergCrashTest") + if (enabled == null || !enabled.equalsIgnoreCase("true") + || crashTestEnabled == null || !crashTestEnabled.equalsIgnoreCase("true")) { + logger.info("disable iceberg crash test") + return + } + + String restPort = context.config.otherConfigs.get("iceberg_rest_uri_port") + String minioPort = context.config.otherConfigs.get("iceberg_minio_port") + String externalEnvIp = context.config.otherConfigs.get("externalEnvIp") + String catalogName = "test_iceberg_write_merge_truncate_negative" + String dbName = "iceberg_write_merge_truncate_negative_db" + + sql """drop catalog if exists ${catalogName}""" + sql """ + create catalog ${catalogName} properties ( + "type" = "iceberg", + "iceberg.catalog.type" = "rest", + "uri" = "http://${externalEnvIp}:${restPort}", + "s3.access_key" = "admin", + "s3.secret_key" = "password", + "s3.endpoint" = "http://${externalEnvIp}:${minioPort}", + "s3.region" = "us-east-1", + "meta.cache.iceberg.table.ttl-second" = "0", + "meta.cache.iceberg.schema.ttl-second" = "0" + ) + """ + sql """switch ${catalogName}""" + sql """drop database if exists ${dbName} force""" + sql """create database ${dbName}""" + sql """use ${dbName}""" + + sql """drop table if exists merge_truncate_negative""" + sql """ + create table merge_truncate_negative ( + id int not null, + partition_value string not null, + payload string + ) + partition by list (truncate(2, partition_value)) () + properties ( + "format-version" = "2", + "write.format.default" = "parquet", + "write.merge.mode" = "merge-on-read", + "write.update.mode" = "merge-on-read" + ) + """ + sql """insert into merge_truncate_negative values (1, 'alpha', 'before')""" + + // WM03-S01: A MERGE source projection is nullable even when every source + // value and the Iceberg target column are NOT NULL. The writer must reject + // an invalid input as a query error and must never terminate a BE. + sql """ + merge into merge_truncate_negative t + using ( + select 1 as id, 'beta' as partition_value, 'after' as payload + union all + select 2, 'gamma', 'inserted' + ) s + on t.id = s.id + when matched then update set + partition_value = s.partition_value, + payload = s.payload + when not matched then + insert (id, partition_value, payload) + values (s.id, s.partition_value, s.payload) + """ + order_qt_merge_truncate_after_fix """ + select id, partition_value, payload + from merge_truncate_negative + order by id + """ + // Logical rows cannot prove the MERGE writer used truncate(2) when routing + // its updated and inserted records. + order_qt_merge_truncate_physical_partitions """ + select distinct hex(struct_element(`partition`, 'partition_value_trunc_2')) + from merge_truncate_negative\$partitions + order by 1 + """ +} diff --git a/regression-test/suites/external_table_p0/iceberg/write/test_iceberg_write_nullability_atomicity.groovy b/regression-test/suites/external_table_p0/iceberg/write/test_iceberg_write_nullability_atomicity.groovy new file mode 100644 index 00000000000000..263134dac3c235 --- /dev/null +++ b/regression-test/suites/external_table_p0/iceberg/write/test_iceberg_write_nullability_atomicity.groovy @@ -0,0 +1,133 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +suite("test_iceberg_write_nullability_atomicity", + "p0,external,iceberg,external_docker,external_docker_iceberg") { + String enabled = context.config.otherConfigs.get("enableIcebergTest") + if (enabled == null || !enabled.equalsIgnoreCase("true")) { + logger.info("disable iceberg test") + return + } + + String restPort = context.config.otherConfigs.get("iceberg_rest_uri_port") + String minioPort = context.config.otherConfigs.get("iceberg_minio_port") + String externalEnvIp = context.config.otherConfigs.get("externalEnvIp") + String catalogName = "test_iceberg_write_nullability_atomicity" + String dbName = "iceberg_write_nullability_atomicity_db" + String internalDb = "iceberg_write_nullability_atomicity_internal_db" + + sql """drop database if exists internal.${internalDb} force""" + sql """create database internal.${internalDb}""" + sql """drop table if exists internal.${internalDb}.nullable_source""" + sql """ + create table internal.${internalDb}.nullable_source ( + id int, + required_text string, + optional_text string + ) + duplicate key(id) + distributed by hash(id) buckets 3 + properties ("replication_num" = "1") + """ + sql """ + insert into internal.${internalDb}.nullable_source values + (2, 'valid-select', null), + (4, 'valid-after-invalid', 'value') + """ + + sql """drop catalog if exists ${catalogName}""" + sql """ + create catalog ${catalogName} properties ( + "type" = "iceberg", + "iceberg.catalog.type" = "rest", + "uri" = "http://${externalEnvIp}:${restPort}", + "s3.access_key" = "admin", + "s3.secret_key" = "password", + "s3.endpoint" = "http://${externalEnvIp}:${minioPort}", + "s3.region" = "us-east-1", + "meta.cache.iceberg.table.ttl-second" = "0", + "meta.cache.iceberg.schema.ttl-second" = "0" + ) + """ + sql """switch ${catalogName}""" + sql """drop database if exists ${dbName} force""" + sql """create database ${dbName}""" + sql """use ${dbName}""" + sql """set enable_fallback_to_original_planner = false""" + + sql """drop table if exists required_sink""" + sql """ + create table required_sink ( + id int not null, + required_text string not null, + optional_text string + ) + partition by list (bucket(8, id)) () + properties ( + "format-version" = "2", + "write.format.default" = "parquet" + ) + """ + sql """insert into required_sink values (1, 'committed', null)""" + + long snapshotsBeforeEvolution = (sql """ + select count(*) from required_sink\$snapshots + """)[0][0] as long + + // W06-S01: Adding a required field or tightening a nullable field is rejected. + test { + sql """alter table required_sink add column new_required int not null""" + exception "doesn't have a default value" + } + test { + sql """alter table required_sink modify column optional_text string not null""" + exception "Can not change nullable column optional_text to not null" + } + assertEquals(snapshotsBeforeEvolution, (sql """ + select count(*) from required_sink\$snapshots + """)[0][0] as long) + + // W06-S02: Distributed and VALUES writes preserve nullable fields while required fields are valid. + sql """ + insert into required_sink + select id, required_text, optional_text + from internal.${internalDb}.nullable_source + """ + sql """insert into required_sink values (5, 'valid-values-retry', null)""" + assertEquals(snapshotsBeforeEvolution + 2, (sql """ + select count(*) from required_sink\$snapshots + """)[0][0] as long) + order_qt_required_after_retry """ + select id, required_text, optional_text + from required_sink + order by id + """ + + sql """refresh table ${dbName}.required_sink""" + spark_iceberg """refresh table demo.${dbName}.required_sink""" + def sparkRows = spark_iceberg """ + select id, required_text, optional_text + from demo.${dbName}.required_sink + order by id + """ + def dorisRows = sql """ + select id, required_text, optional_text + from required_sink + order by id + """ + assertSparkDorisResultEquals(sparkRows, dorisRows) +} diff --git a/regression-test/suites/external_table_p0/iceberg/write/test_iceberg_write_nullable_truncate_negative.groovy b/regression-test/suites/external_table_p0/iceberg/write/test_iceberg_write_nullable_truncate_negative.groovy new file mode 100644 index 00000000000000..25074a0af78d63 --- /dev/null +++ b/regression-test/suites/external_table_p0/iceberg/write/test_iceberg_write_nullable_truncate_negative.groovy @@ -0,0 +1,97 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +suite("test_iceberg_write_nullable_truncate_negative", + "p0,external,iceberg,external_docker,external_docker_iceberg") { + String enabled = context.config.otherConfigs.get("enableIcebergTest") + if (enabled == null || !enabled.equalsIgnoreCase("true")) { + logger.info("disable iceberg test") + return + } + + // This opt-in switch isolates a BE-fatal negative scenario from the shared P0 cluster. + // Enable it only in a cluster whose BE processes can be restarted after the suite. + String crashTestEnabled = context.config.otherConfigs.get("enableIcebergCrashTest") + if (crashTestEnabled == null || !crashTestEnabled.equalsIgnoreCase("true")) { + logger.info("skip isolated Iceberg crash regression") + return + } + + String restPort = context.config.otherConfigs.get("iceberg_rest_uri_port") + String minioPort = context.config.otherConfigs.get("iceberg_minio_port") + String externalEnvIp = context.config.otherConfigs.get("externalEnvIp") + String catalogName = "test_iceberg_write_nullable_truncate_negative" + String dbName = "iceberg_write_nullable_truncate_negative_db" + + sql """drop catalog if exists ${catalogName}""" + sql """ + create catalog ${catalogName} properties ( + "type" = "iceberg", + "iceberg.catalog.type" = "rest", + "uri" = "http://${externalEnvIp}:${restPort}", + "s3.access_key" = "admin", + "s3.secret_key" = "password", + "s3.endpoint" = "http://${externalEnvIp}:${minioPort}", + "s3.region" = "us-east-1" + ) + """ + sql """switch ${catalogName}""" + sql """drop database if exists ${dbName} force""" + sql """create database ${dbName}""" + sql """use ${dbName}""" + + sql """ + create table nullable_truncate ( + id int not null, + zone string + ) + partition by list (zone) () + properties ( + "format-version" = "2", + "write.delete.mode" = "merge-on-read", + "write.update.mode" = "merge-on-read", + "write.merge.mode" = "merge-on-read" + ) + """ + sql """insert into nullable_truncate values (1, 'CN'), (2, null)""" + + // Negative scenario: evolve to a truncate transform whose source remains nullable, + // then write both non-NULL and NULL partition values through Doris. + sql """ + alter table nullable_truncate + add partition key truncate(2, zone) as zone_prefix + """ + sql """insert into nullable_truncate values (3, 'US-east'), (4, null)""" + // UPDATE creates a Nullable projection even though the non-NULL branch is + // selected for id 3; the partition transformer must preserve that wrapper. + sql """ + update nullable_truncate + set zone = if(id = 3, 'US-west', cast(null as string)) + where id in (3, 4) + """ + + order_qt_nullable_truncate_rows """ + select id, zone from nullable_truncate order by id + """ + order_qt_nullable_truncate_physical_partitions """ + select distinct spec_id, + hex(struct_element(`partition`, 'zone')), + hex(struct_element(`partition`, 'zone_prefix')) + from nullable_truncate\$partitions + order by spec_id, 2, 3 + """ +} diff --git a/regression-test/suites/external_table_p0/iceberg/write/test_iceberg_write_order_distribution_properties.groovy b/regression-test/suites/external_table_p0/iceberg/write/test_iceberg_write_order_distribution_properties.groovy new file mode 100644 index 00000000000000..41e8b5a199ef4e --- /dev/null +++ b/regression-test/suites/external_table_p0/iceberg/write/test_iceberg_write_order_distribution_properties.groovy @@ -0,0 +1,215 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +suite("test_iceberg_write_order_distribution_properties", + "p0,external,iceberg,external_docker,external_docker_iceberg") { + String enabled = context.config.otherConfigs.get("enableIcebergTest") + if (enabled == null || !enabled.equalsIgnoreCase("true")) { + logger.info("disable iceberg test") + return + } + + String restPort = context.config.otherConfigs.get("iceberg_rest_uri_port") + String minioPort = context.config.otherConfigs.get("iceberg_minio_port") + String externalEnvIp = context.config.otherConfigs.get("externalEnvIp") + String catalogName = "test_iceberg_write_order_distribution_properties" + String dbName = "iceberg_write_order_distribution_properties_db" + + sql """drop catalog if exists ${catalogName}""" + sql """ + create catalog ${catalogName} properties ( + "type" = "iceberg", + "iceberg.catalog.type" = "rest", + "uri" = "http://${externalEnvIp}:${restPort}", + "s3.access_key" = "admin", + "s3.secret_key" = "password", + "s3.endpoint" = "http://${externalEnvIp}:${minioPort}", + "s3.region" = "us-east-1", + "meta.cache.iceberg.table.ttl-second" = "0", + "meta.cache.iceberg.schema.ttl-second" = "0" + ) + """ + sql """switch ${catalogName}""" + sql """drop database if exists ${dbName} force""" + sql """create database ${dbName}""" + sql """use ${dbName}""" + + sql """drop table if exists ordered_evolution""" + sql """ + create table ordered_evolution ( + id int, + region string, + payload string, + score int + ) + order by (region asc nulls last, id desc nulls first) + partition by list (region) () + properties ( + "format-version" = "2", + "write.format.default" = "parquet", + "write.delete.mode" = "merge-on-read", + "write.update.mode" = "merge-on-read", + "write.merge.mode" = "merge-on-read", + "write.distribution-mode" = "range" + ) + """ + + // WP01-S01: The planned Iceberg write contains the declared global order, + // including direction and NULL ordering. + explain { + sql """ + insert into ordered_evolution + select number, + if(number % 4 = 0, null, concat('R', number % 3)), + concat('payload-', number), + number + from numbers('number' = '32') + """ + contains "ORDER BY (`region` ASC NULLS LAST, `id` DESC NULLS FIRST)" + } + + // WP01-S02: Force several sorted writer flushes on a distributed source. + // This exercises global order, NULL partitions and file rollover together. + sql """set iceberg_write_target_file_size_bytes = 51200""" + sql """ + insert into ordered_evolution + select number, + if(number % 7 = 0, null, concat('R', number % 5)), + concat('payload-', number, '-', repeat('x', 64)), + number + from numbers('number' = '10000') + """ + def filesAfterInsert = sql """ + select count(*), sum(record_count) + from ordered_evolution\$files + """ + assertEquals(10000L, filesAfterInsert[0][1] as long) + def maxFilesPerPartition = sql """ + select max(file_count) + from ( + select `partition`, count(*) as file_count + from ordered_evolution\$files + group by `partition` + ) partition_file_counts + """ + // Partition fan-out already creates multiple files globally, so rollover + // is proven only when one physical partition owns more than one file. + assertTrue((maxFilesPerPartition[0][0] as long) > 1L) + + // WP01-S03: Schema evolution and row-level DML continue to use the current + // sort order and preserve Spark/Doris visible results. + sql """alter table ordered_evolution add column status string""" + sql """ + update ordered_evolution + set region = 'R-updated', score = score + 10000, status = 'updated' + where id in (1, 7) + """ + sql """ + merge into ordered_evolution t + using ( + select 2 as id, cast(null as string) as region, 'merge-update' as payload, + 20002 as score, 'U' as op + union all + select 10001, 'R-new', 'merge-insert', 10001, 'I' + ) s + on t.id = s.id + when matched then update set + region = s.region, + payload = s.payload, + score = s.score, + status = 'merged' + when not matched then + insert (id, region, payload, score, status) + values (s.id, s.region, s.payload, s.score, 'inserted') + """ + order_qt_ordered_evolution_changed_rows """ + select id, region, payload, score, status + from ordered_evolution + where id in (1, 2, 7, 10001) + order by id + """ + order_qt_ordered_evolution_files """ + select lower(file_format), sum(record_count) + from ordered_evolution\$files + group by lower(file_format) + order by lower(file_format) + """ + spark_iceberg """refresh table demo.${dbName}.ordered_evolution""" + def sparkRows = spark_iceberg """ + select id, region, payload, score, status + from demo.${dbName}.ordered_evolution + where id in (1, 2, 7, 10001) + order by id + """ + def dorisRows = sql """ + select id, region, payload, score, status + from ordered_evolution + where id in (1, 2, 7, 10001) + order by id + """ + assertSparkDorisResultEquals(sparkRows, dorisRows) + sql """set iceberg_write_target_file_size_bytes = 0""" + + // WP01-S04: All official distribution-mode property values remain + // correctness-compatible with Doris distributed writes. + for (String mode : ["none", "hash", "range"]) { + String tableName = "distribution_${mode}" + sql """drop table if exists ${tableName}""" + sql """ + create table ${tableName} ( + id int, + region string, + payload string + ) + partition by list (region, bucket(8, id)) () + properties ( + "format-version" = "2", + "write.format.default" = "orc", + "write.distribution-mode" = "${mode}" + ) + """ + sql """ + insert into ${tableName} + select number, + if(number % 11 = 0, null, concat('R', number % 9)), + concat('${mode}-', number) + from numbers('number' = '512') + """ + def distributionRows = sql """select count(*), count(distinct id) from ${tableName}""" + assertEquals(512L, distributionRows[0][0] as long) + assertEquals(512L, distributionRows[0][1] as long) + def sparkDistributionRows = spark_iceberg """ + select id, region, payload + from demo.${dbName}.${tableName} + order by id + """ + def dorisDistributionRows = sql """ + select id, region, payload + from ${tableName} + order by id + """ + assertSparkDorisResultEquals(sparkDistributionRows, dorisDistributionRows) + } + order_qt_distribution_mode_counts """ + select 'hash', count(*) from distribution_hash + union all + select 'none', count(*) from distribution_none + union all + select 'range', count(*) from distribution_range + order by 1 + """ +} diff --git a/regression-test/suites/external_table_p0/iceberg/write/test_iceberg_write_overwrite_atomicity.groovy b/regression-test/suites/external_table_p0/iceberg/write/test_iceberg_write_overwrite_atomicity.groovy new file mode 100644 index 00000000000000..fe6d8f2cc3799a --- /dev/null +++ b/regression-test/suites/external_table_p0/iceberg/write/test_iceberg_write_overwrite_atomicity.groovy @@ -0,0 +1,163 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +suite("test_iceberg_write_overwrite_atomicity", + "p0,external,iceberg,external_docker,external_docker_iceberg") { + String enabled = context.config.otherConfigs.get("enableIcebergTest") + if (enabled == null || !enabled.equalsIgnoreCase("true")) { + logger.info("disable iceberg test") + return + } + + String restPort = context.config.otherConfigs.get("iceberg_rest_uri_port") + String minioPort = context.config.otherConfigs.get("iceberg_minio_port") + String externalEnvIp = context.config.otherConfigs.get("externalEnvIp") + String catalogName = "test_iceberg_write_overwrite_atomicity" + String dbName = "iceberg_write_overwrite_atomicity_db" + + sql """drop catalog if exists ${catalogName}""" + sql """ + create catalog ${catalogName} properties ( + "type" = "iceberg", + "iceberg.catalog.type" = "rest", + "uri" = "http://${externalEnvIp}:${restPort}", + "s3.access_key" = "admin", + "s3.secret_key" = "password", + "s3.endpoint" = "http://${externalEnvIp}:${minioPort}", + "s3.region" = "us-east-1" + ) + """ + sql """switch ${catalogName}""" + sql """drop database if exists ${dbName} force""" + sql """create database ${dbName}""" + sql """use ${dbName}""" + + sql """drop table if exists overwrite_atomicity""" + sql """ + create table overwrite_atomicity ( + id int not null, + region string, + payload string + ) + partition by list (region) () + properties ( + "format-version" = "2", + "write.format.default" = "parquet" + ) + """ + sql """ + insert into overwrite_atomicity values + (1, 'A', 'committed-a'), + (2, 'B', 'committed-b') + """ + sql """alter table overwrite_atomicity create branch retry_branch""" + sql """set enable_strict_cast = true""" + + // WO03-S01: A distributed expression failure must not publish a partial + // overwrite, remove an existing partition, or create a new snapshot. + long snapshotsBeforeFailure = + (sql """select count(*) from overwrite_atomicity\$snapshots""")[0][0] as long + long filesBeforeFailure = + (sql """select count(*) from overwrite_atomicity\$files""")[0][0] as long + test { + sql """ + insert overwrite table overwrite_atomicity + select cast(if(number = 2, 'invalid-id', cast(number + 10 as string)) as int), + if(number % 2 = 0, 'A', 'C'), + concat('candidate-', number) + from numbers('number' = '8') + """ + exception "can't cast to INT in strict mode" + } + assertEquals(snapshotsBeforeFailure, + (sql """select count(*) from overwrite_atomicity\$snapshots""")[0][0] as long) + assertEquals(filesBeforeFailure, + (sql """select count(*) from overwrite_atomicity\$files""")[0][0] as long) + order_qt_overwrite_failure_state """ + select id, region, payload + from overwrite_atomicity + order by id + """ + + // WO03-S02: The same invariant applies to a branch-qualified overwrite. + test { + sql """ + insert overwrite table overwrite_atomicity@branch(retry_branch) + select cast(if(number = 3, 'invalid-id', cast(number + 20 as string)) as int), + 'A', + concat('branch-candidate-', number) + from numbers('number' = '8') + """ + exception "can't cast to INT in strict mode" + } + order_qt_branch_overwrite_failure_state """ + select id, region, payload + from overwrite_atomicity@branch(retry_branch) + order by id + """ + order_qt_main_after_branch_overwrite_failure """ + select id, region, payload + from overwrite_atomicity + order by id + """ + // A branch failure is only atomic if both engines still observe the same + // pre-write branch state; a Doris-only query could hide unreadable files. + spark_iceberg """refresh table demo.${dbName}.overwrite_atomicity""" + def sparkBranchRows = spark_iceberg """ + select id, region, payload + from demo.${dbName}.overwrite_atomicity version as of 'retry_branch' + order by id + """ + def dorisBranchRows = sql """ + select id, region, payload + from overwrite_atomicity@branch(retry_branch) + order by id + """ + assertSparkDorisResultEquals(sparkBranchRows, dorisBranchRows) + + // WO03-S03: Retry the corrected logical operation. Each replacement row + // becomes visible exactly once and only one new main snapshot is committed. + sql """ + insert overwrite table overwrite_atomicity + select number + 10, + if(number % 2 = 0, 'A', 'C'), + concat('candidate-', number) + from numbers('number' = '8') + """ + assertEquals(snapshotsBeforeFailure + 1, + (sql """select count(*) from overwrite_atomicity\$snapshots""")[0][0] as long) + order_qt_overwrite_retry """ + select id, region, payload, count(*) + from overwrite_atomicity + group by id, region, payload + order by id + """ + // Refresh after the retry so cross-engine validation covers the corrected + // overwrite commit, not merely the failure baselines. + spark_iceberg """refresh table demo.${dbName}.overwrite_atomicity""" + def sparkRows = spark_iceberg """ + select id, region, payload + from demo.${dbName}.overwrite_atomicity + order by id + """ + def dorisRows = sql """ + select id, region, payload + from overwrite_atomicity + order by id + """ + assertSparkDorisResultEquals(sparkRows, dorisRows) +} diff --git a/regression-test/suites/external_table_p0/iceberg/write/test_iceberg_write_overwrite_delete_files.groovy b/regression-test/suites/external_table_p0/iceberg/write/test_iceberg_write_overwrite_delete_files.groovy new file mode 100644 index 00000000000000..d787962094a72a --- /dev/null +++ b/regression-test/suites/external_table_p0/iceberg/write/test_iceberg_write_overwrite_delete_files.groovy @@ -0,0 +1,197 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +suite("test_iceberg_write_overwrite_delete_files", + "p0,external,iceberg,external_docker,external_docker_iceberg") { + String enabled = context.config.otherConfigs.get("enableIcebergTest") + if (enabled == null || !enabled.equalsIgnoreCase("true")) { + logger.info("disable iceberg test") + return + } + + String restPort = context.config.otherConfigs.get("iceberg_rest_uri_port") + String minioPort = context.config.otherConfigs.get("iceberg_minio_port") + String externalEnvIp = context.config.otherConfigs.get("externalEnvIp") + String catalogName = "test_iceberg_write_overwrite_delete_files" + String dbName = "iceberg_write_overwrite_delete_files_db" + + def assertSparkMatchesDoris = { + sql """refresh table ${dbName}.overwrite_delete_files""" + spark_iceberg """refresh table demo.${dbName}.overwrite_delete_files""" + def sparkRows = spark_iceberg """ + select id, region, bucket_key, payload + from demo.${dbName}.overwrite_delete_files + order by id + """ + def dorisRows = sql """ + select id, region, bucket_key, payload + from overwrite_delete_files + order by id + """ + assertSparkDorisResultEquals(sparkRows, dorisRows) + } + + sql """drop catalog if exists ${catalogName}""" + sql """ + create catalog ${catalogName} properties ( + "type" = "iceberg", + "iceberg.catalog.type" = "rest", + "uri" = "http://${externalEnvIp}:${restPort}", + "s3.access_key" = "admin", + "s3.secret_key" = "password", + "s3.endpoint" = "http://${externalEnvIp}:${minioPort}", + "s3.region" = "us-east-1", + "meta.cache.iceberg.table.ttl-second" = "0", + "meta.cache.iceberg.schema.ttl-second" = "0" + ) + """ + sql """switch ${catalogName}""" + sql """drop database if exists ${dbName} force""" + sql """create database ${dbName}""" + sql """use ${dbName}""" + + sql """drop table if exists overwrite_delete_files""" + sql """ + create table overwrite_delete_files ( + id int not null, + region string, + bucket_key string not null, + payload string + ) + partition by list (region, bucket(4, bucket_key)) () + properties ( + "format-version" = "2", + "write.format.default" = "parquet", + "write.delete.mode" = "merge-on-read", + "write.update.mode" = "merge-on-read", + "write.merge.mode" = "merge-on-read" + ) + """ + sql """ + insert into overwrite_delete_files values + (1, 'A', 'alpha', 'keep-a'), + (2, 'A', 'beta', 'delete-a'), + (3, 'B', 'gamma', 'move-b-to-c'), + (4, 'B', 'delta', 'merge-delete-b'), + (5, null, 'null-key', 'keep-null') + """ + String baseSnapshot = (sql """ + select snapshot_id from overwrite_delete_files\$snapshots + order by committed_at desc limit 1 + """)[0][0].toString() + sql """alter table overwrite_delete_files create tag before_row_dml as of version ${baseSnapshot}""" + + // WO02-S01: Generate position deletes in several partitions and move one + // updated row to a new partition before overwrite. + sql """delete from overwrite_delete_files where id = 2""" + sql """ + update overwrite_delete_files + set region = 'C', bucket_key = 'gamma-new', payload = 'moved-to-c' + where id = 3 + """ + sql """ + merge into overwrite_delete_files t + using ( + select 4 as id, 'D' as region, 'delta-new' as bucket_key, + 'delete' as payload, 'D' as op + union all + select 6, 'B', 'echo', 'merge-insert-b', 'I' + ) s + on t.id = s.id + when matched and s.op = 'D' then delete + when not matched then + insert (id, region, bucket_key, payload) + values (s.id, s.region, s.bucket_key, s.payload) + """ + order_qt_before_overwrite_rows """ + select id, region, bucket_key, payload + from overwrite_delete_files + order by id + """ + order_qt_before_overwrite_delete_files """ + select spec_id, sum(record_count) + from overwrite_delete_files\$delete_files + group by spec_id + order by spec_id + """ + + // WO02-S02: Overwrite only current partitions produced by the input. Delete + // files that refer to replaced data must not hide the replacement rows. + sql """ + insert overwrite table overwrite_delete_files + values + (10, 'A', 'alpha', 'replacement-a'), + (11, 'B', 'echo', 'replacement-b') + """ + order_qt_after_overwrite_rows """ + select id, region, bucket_key, payload + from overwrite_delete_files + order by id + """ + order_qt_after_overwrite_delete_files """ + select spec_id, sum(record_count) + from overwrite_delete_files\$delete_files + group by spec_id + order by spec_id + """ + order_qt_before_row_dml_tag """ + select id, region, bucket_key, payload + from overwrite_delete_files@tag(before_row_dml) + order by id + """ + assertSparkMatchesDoris() + + // WO02-S03: Repeat after partition evolution so old-spec delete files and + // current-spec replacements coexist without leaking across specs. + sql """ + alter table overwrite_delete_files + replace partition key bucket(4, bucket_key) + with bucket(8, bucket_key) as bucket_key_8 + """ + sql """ + alter table overwrite_delete_files + add partition key truncate(1, bucket_key) as bucket_key_prefix + """ + sql """ + insert into overwrite_delete_files values + (12, 'A', 'alpha-new', 'new-spec-a'), + (13, null, 'null-new', 'new-spec-null') + """ + sql """delete from overwrite_delete_files where id = 12""" + sql """ + insert overwrite table overwrite_delete_files + values (14, 'A', 'alpha-new', 'new-spec-replacement-a') + """ + order_qt_evolved_overwrite_rows """ + select id, region, bucket_key, payload + from overwrite_delete_files + order by id + """ + order_qt_evolved_overwrite_specs """ + select spec_id, count(*), sum(record_count) + from overwrite_delete_files\$partitions + group by spec_id + order by spec_id + """ + order_qt_evolved_overwrite_delete_files """ + select spec_id, sum(record_count) + from overwrite_delete_files\$delete_files + group by spec_id + order by spec_id + """ + assertSparkMatchesDoris() +} diff --git a/regression-test/suites/external_table_p0/iceberg/write/test_iceberg_write_overwrite_evolution.groovy b/regression-test/suites/external_table_p0/iceberg/write/test_iceberg_write_overwrite_evolution.groovy new file mode 100644 index 00000000000000..07fa302ba52193 --- /dev/null +++ b/regression-test/suites/external_table_p0/iceberg/write/test_iceberg_write_overwrite_evolution.groovy @@ -0,0 +1,180 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +suite("test_iceberg_write_overwrite_evolution", + "p0,external,iceberg,external_docker,external_docker_iceberg") { + String enabled = context.config.otherConfigs.get("enableIcebergTest") + if (enabled == null || !enabled.equalsIgnoreCase("true")) { + logger.info("disable iceberg test") + return + } + + String restPort = context.config.otherConfigs.get("iceberg_rest_uri_port") + String minioPort = context.config.otherConfigs.get("iceberg_minio_port") + String externalEnvIp = context.config.otherConfigs.get("externalEnvIp") + String catalogName = "test_iceberg_write_overwrite_evolution" + String dbName = "iceberg_write_overwrite_evolution_db" + + def assertSparkMatchesDoris = { + sql """refresh table ${dbName}.overwrite_evolution""" + spark_iceberg """refresh table demo.${dbName}.overwrite_evolution""" + def sparkRows = spark_iceberg """ + select id, region, code, event_time, payload + from demo.${dbName}.overwrite_evolution + order by id + """ + def dorisRows = sql """ + select id, region, code, event_time, payload + from overwrite_evolution + order by id + """ + assertSparkDorisResultEquals(sparkRows, dorisRows) + } + + sql """drop catalog if exists ${catalogName}""" + sql """ + create catalog ${catalogName} properties ( + "type" = "iceberg", + "iceberg.catalog.type" = "rest", + "uri" = "http://${externalEnvIp}:${restPort}", + "s3.access_key" = "admin", + "s3.secret_key" = "password", + "s3.endpoint" = "http://${externalEnvIp}:${minioPort}", + "s3.region" = "us-east-1", + "meta.cache.iceberg.table.ttl-second" = "0", + "meta.cache.iceberg.schema.ttl-second" = "0" + ) + """ + sql """switch ${catalogName}""" + sql """drop database if exists ${dbName} force""" + sql """create database ${dbName}""" + sql """use ${dbName}""" + + sql """drop table if exists overwrite_evolution""" + sql """ + create table overwrite_evolution ( + id int not null, + region string, + code string not null, + event_time datetime, + payload string + ) + partition by list (region, bucket(4, code), day(event_time)) () + properties ( + "format-version" = "2", + "write.format.default" = "parquet" + ) + """ + + // WO01-S01: Build old-spec files and protect the baseline with both a tag + // and a branch before changing the partition granularity. + sql """ + insert into overwrite_evolution values + (1, 'A', 'alpha', '2026-01-01 01:10:00', 'old-hour-1'), + (2, 'A', 'beta', '2026-01-01 02:20:00', 'old-hour-2'), + (3, null, 'null-region', null, 'old-null'), + (4, 'B', 'delta', '2026-01-02 01:30:00', 'old-other-day') + """ + String baseSnapshot = (sql """ + select snapshot_id from overwrite_evolution\$snapshots + order by committed_at desc limit 1 + """)[0][0].toString() + sql """alter table overwrite_evolution create tag overwrite_base as of version ${baseSnapshot}""" + sql """alter table overwrite_evolution create branch overwrite_audit as of version ${baseSnapshot}""" + + // WO01-S02: Keep day(event_time), add hour(event_time), replace the STRING + // bucket and add STRING truncate. Old and new specs must remain independently visible. + sql """alter table overwrite_evolution add partition key hour(event_time) as event_hour""" + sql """ + alter table overwrite_evolution + replace partition key bucket(4, code) with bucket(8, code) as code_bucket_8 + """ + sql """alter table overwrite_evolution add partition key truncate(2, code) as code_prefix""" + sql """ + insert into overwrite_evolution values + (5, 'A', 'alpha-new', '2026-01-01 01:40:00', 'new-spec-before-overwrite'), + (6, null, 'null-new', null, 'new-spec-null'), + (7, 'C', 'charlie', '2026-02-01 03:00:00', 'new-spec-other-month') + """ + + // WO01-S03: Dynamic overwrite operates on current-spec partitions. It must + // not silently remove old day-level files that cannot be equal to a new spec. + sql """ + insert overwrite table overwrite_evolution + values (8, 'A', 'alpha-new', '2026-01-01 01:40:00', 'overwrite-current-spec') + """ + order_qt_overwrite_current """ + select id, region, code, event_time, payload + from overwrite_evolution + order by id + """ + order_qt_overwrite_specs """ + select spec_id, count(*), sum(record_count) + from overwrite_evolution\$partitions + group by spec_id + order by spec_id + """ + order_qt_overwrite_base_tag """ + select id, region, code, event_time, payload + from overwrite_evolution@tag(overwrite_base) + order by id + """ + order_qt_overwrite_audit_branch """ + select id, region, code, event_time, payload + from overwrite_evolution@branch(overwrite_audit) + order by id + """ + assertSparkMatchesDoris() + + // WO01-S04: Seed the complete post-evolution partition tuple before its + // overwrite; reusing id 9 also fixes the new bucket value across both writes. + sql """alter table overwrite_evolution drop partition key region""" + sql """alter table overwrite_evolution add partition key bucket(4, id) as id_bucket""" + sql """ + insert into overwrite_evolution + values (9, null, 'null-new', null, 'overwrite-seed-null-current-spec') + """ + assertEquals(1L, (sql """ + select count(*) from overwrite_evolution + where payload = 'overwrite-seed-null-current-spec' + """)[0][0] as long) + sql """ + insert overwrite table overwrite_evolution + values (9, null, 'null-new', null, 'overwrite-null-current-spec') + """ + assertEquals(0L, (sql """ + select count(*) from overwrite_evolution + where payload = 'overwrite-seed-null-current-spec' + """)[0][0] as long) + order_qt_overwrite_after_drop_identity """ + select id, region, code, event_time, payload + from overwrite_evolution + order by id + """ + order_qt_overwrite_after_drop_identity_specs """ + select spec_id, count(*), sum(record_count) + from overwrite_evolution\$partitions + group by spec_id + order by spec_id + """ + order_qt_overwrite_base_tag_after_second_evolution """ + select id, region, code, event_time, payload + from overwrite_evolution@tag(overwrite_base) + order by id + """ + assertSparkMatchesDoris() +} diff --git a/regression-test/suites/external_table_p0/iceberg/write/test_iceberg_write_partition_types_null.groovy b/regression-test/suites/external_table_p0/iceberg/write/test_iceberg_write_partition_types_null.groovy new file mode 100644 index 00000000000000..aeb8614d5bebfa --- /dev/null +++ b/regression-test/suites/external_table_p0/iceberg/write/test_iceberg_write_partition_types_null.groovy @@ -0,0 +1,287 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +suite("test_iceberg_write_partition_types_null", + "p0,external,iceberg,external_docker,external_docker_iceberg") { + String enabled = context.config.otherConfigs.get("enableIcebergTest") + if (enabled == null || !enabled.equalsIgnoreCase("true")) { + logger.info("disable iceberg test") + return + } + + String restPort = context.config.otherConfigs.get("iceberg_rest_uri_port") + String minioPort = context.config.otherConfigs.get("iceberg_minio_port") + String externalEnvIp = context.config.otherConfigs.get("externalEnvIp") + String catalogName = "test_iceberg_write_partition_types_null" + String dbName = "iceberg_write_partition_types_null_db" + + def assertSparkMatchesDoris = { String tableName, String projection -> + sql """refresh table ${dbName}.${tableName}""" + spark_iceberg """refresh table demo.${dbName}.${tableName}""" + def sparkRows = spark_iceberg """ + select ${projection} + from demo.${dbName}.${tableName} + order by id + """ + def dorisRows = sql """ + select ${projection} + from ${tableName} + order by id + """ + assertSparkDorisResultEquals(sparkRows, dorisRows) + } + + sql """drop catalog if exists ${catalogName}""" + sql """ + create catalog ${catalogName} properties ( + "type" = "iceberg", + "iceberg.catalog.type" = "rest", + "uri" = "http://${externalEnvIp}:${restPort}", + "s3.access_key" = "admin", + "s3.secret_key" = "password", + "s3.endpoint" = "http://${externalEnvIp}:${minioPort}", + "s3.region" = "us-east-1", + "meta.cache.iceberg.table.ttl-second" = "0", + "meta.cache.iceberg.schema.ttl-second" = "0" + ) + """ + sql """switch ${catalogName}""" + sql """drop database if exists ${dbName} force""" + sql """create database ${dbName}""" + sql """use ${dbName}""" + sql """set enable_fallback_to_original_planner = false""" + + // W05-S00: BOOLEAN is valid for identity but not for Iceberg's bucket transform. + // The invalid table must be rejected instead of creating a table that fails on its first write. + test { + sql """ + create table invalid_boolean_bucket ( + id int, + p_bool boolean + ) + partition by list (bucket(4, p_bool)) () + """ + exception "Invalid source type boolean for transform: bucket[4]" + } + + // W05-S01: STRING supports identity, bucket and truncate together. + // NULL is routed by the nullable identity source while transform-specific sources stay required. + sql """drop table if exists string_partitions""" + sql """ + create table string_partitions ( + id int not null, + p_string string, + p_bucket string not null, + p_truncate string not null, + payload string + ) + partition by list (p_string, bucket(8, p_bucket), truncate(2, p_truncate)) () + properties ( + "format-version" = "2", + "write.format.default" = "parquet" + ) + """ + sql """ + insert into string_partitions values + (1, 'alpha', 'bucket-a', 'alpha', 'a'), + (2, 'alphabet', 'bucket-b', 'alphabet', 'ab'), + (3, '', 'bucket-empty', '', 'empty'), + (4, '中文', 'bucket-unicode', '中文', 'unicode'), + (5, null, 'bucket-null-identity', 'null-identity', 'null-string') + """ + order_qt_string_rows """ + select id, p_string, p_bucket, p_truncate, payload + from string_partitions + order by id + """ + order_qt_string_null_filter """ + select id from string_partitions where p_string is null order by id + """ + + // W05-S02: Replace a STRING bucket transform and keep old/new specs filterable. + sql """ + alter table string_partitions + replace partition key bucket(8, p_bucket) + with bucket(16, p_bucket) as p_string_bucket_16 + """ + sql """ + insert into string_partitions values + (6, 'beta', 'bucket-new', 'beta', 'new-spec'), + (7, null, 'bucket-new-null-identity', 'null-identity', 'new-null-string') + """ + order_qt_string_cross_spec_filter """ + select id from string_partitions + where p_string is null or p_string like 'alp%' + order by id + """ + // Filtering the replaced bucket source exercises both bucket widths; + // predicates on the unchanged identity source cannot detect a bad bucket. + order_qt_string_cross_spec_bucket_filter """ + select id from string_partitions + where p_bucket in ('bucket-a', 'bucket-new') + order by id + """ + order_qt_string_partition_specs """ + select spec_id, sum(record_count) + from string_partitions\$partitions + group by spec_id + order by spec_id + """ + assertSparkMatchesDoris( + "string_partitions", + "id, p_string, p_bucket, p_truncate, payload") + + // W05-S03: Integer/BIGINT/DECIMAL bucket or truncate transforms and BOOLEAN identity + // must all route NULL to valid Iceberg partitions. + sql """drop table if exists numeric_partitions""" + sql """ + create table numeric_partitions ( + id int not null, + p_int int, + p_bigint bigint, + p_decimal decimal(12, 2), + p_bool boolean, + payload string + ) + partition by list ( + bucket(4, p_int), + bucket(8, p_bigint), + truncate(100, p_bigint), + bucket(8, p_decimal), + truncate(10, p_decimal), + p_bool + ) () + properties ( + "format-version" = "2", + "write.format.default" = "orc" + ) + """ + sql """ + insert into numeric_partitions values + (1, 1, 101, 11.11, true, 'positive'), + (2, -1, -101, -11.11, false, 'negative'), + (3, 0, 0, 0.00, null, 'zero-null-bool'), + (4, null, null, null, null, 'all-null') + """ + order_qt_numeric_rows """ + select id, p_int, p_bigint, p_decimal, p_bool, payload + from numeric_partitions + order by id + """ + order_qt_numeric_null_filter """ + select id from numeric_partitions + where p_int is null or p_bigint is null or p_decimal is null or p_bool is null + order by id + """ + order_qt_numeric_partitions """ + select spec_id, sum(record_count) + from numeric_partitions\$partitions + group by spec_id + order by spec_id + """ + // Logical rows and spec totals do not expose incorrect transform values. + order_qt_numeric_physical_partitions """ + select struct_element(`partition`, 'p_int_bucket'), + struct_element(`partition`, 'p_bigint_bucket'), + struct_element(`partition`, 'p_bigint_trunc'), + struct_element(`partition`, 'p_decimal_bucket'), + struct_element(`partition`, 'p_decimal_trunc'), + struct_element(`partition`, 'p_bool'), + record_count + from numeric_partitions\$partitions + order by 1, 2, 3, 4, 5, 6 + """ + assertSparkMatchesDoris( + "numeric_partitions", + "id, p_int, p_bigint, p_decimal, p_bool, payload") + + // W05-S04: DATE/DATETIME time transforms accept boundary values and NULL. + sql """drop table if exists temporal_partitions""" + sql """ + create table temporal_partitions ( + id int not null, + p_date_bucket date, + p_date_year date, + p_date_month date, + p_ts_bucket datetime, + p_ts_day datetime, + p_ts_hour datetime, + payload string + ) + partition by list ( + bucket(8, p_date_bucket), + year(p_date_year), + month(p_date_month), + bucket(8, p_ts_bucket), + day(p_ts_day), + hour(p_ts_hour) + ) () + properties ( + "format-version" = "2", + "write.format.default" = "parquet" + ) + """ + sql """ + insert into temporal_partitions values + (1, '1969-12-31', '1969-12-31', '1969-12-31', + '1969-12-31 23:59:59', '1969-12-31 23:59:59', '1969-12-31 23:59:59', + 'before-epoch'), + (2, '1970-01-01', '1970-01-01', '1970-01-01', + '1970-01-01 00:00:00', '1970-01-01 00:00:00', '1970-01-01 00:00:00', + 'epoch'), + (3, '2024-02-29', '2024-02-29', '2024-02-29', + '2024-02-29 12:34:56', '2024-02-29 12:34:56', '2024-02-29 12:34:56', + 'leap-day'), + (4, null, null, null, null, null, null, 'all-null') + """ + order_qt_temporal_rows """ + select id, p_date_bucket, p_date_year, p_date_month, + p_ts_bucket, p_ts_day, p_ts_hour, payload + from temporal_partitions + order by id + """ + order_qt_temporal_filters """ + select id from temporal_partitions + where p_date_bucket is null + or p_ts_hour < timestamp '1970-01-01 00:00:00' + or p_date_month = date '2024-02-29' + order by id + """ + order_qt_temporal_partitions """ + select spec_id, sum(record_count) + from temporal_partitions\$partitions + group by spec_id + order by spec_id + """ + // Read each time-transform field from Iceberg metadata so epoch boundaries + // and NULL routing are observable independently of source-row equality. + order_qt_temporal_physical_partitions """ + select struct_element(`partition`, 'p_date_bucket_bucket'), + struct_element(`partition`, 'p_date_year_year'), + struct_element(`partition`, 'p_date_month_month'), + struct_element(`partition`, 'p_ts_bucket_bucket'), + struct_element(`partition`, 'p_ts_day_day'), + struct_element(`partition`, 'p_ts_hour_hour'), + record_count + from temporal_partitions\$partitions + order by 1, 2, 3, 4, 5, 6 + """ + assertSparkMatchesDoris( + "temporal_partitions", + "id, p_date_bucket, p_date_year, p_date_month, " + + "p_ts_bucket, p_ts_day, p_ts_hour, payload") +} diff --git a/regression-test/suites/external_table_p0/iceberg/write/test_iceberg_write_required_null_select_negative.groovy b/regression-test/suites/external_table_p0/iceberg/write/test_iceberg_write_required_null_select_negative.groovy new file mode 100644 index 00000000000000..765e5c58345dae --- /dev/null +++ b/regression-test/suites/external_table_p0/iceberg/write/test_iceberg_write_required_null_select_negative.groovy @@ -0,0 +1,103 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +suite("test_iceberg_write_required_null_select_negative", + "p0,external,iceberg,external_docker,external_docker_iceberg") { + String enabled = context.config.otherConfigs.get("enableIcebergTest") + if (enabled == null || !enabled.equalsIgnoreCase("true")) { + logger.info("disable iceberg test") + return + } + + // This opt-in switch isolates a write that can publish an unreadable Iceberg data file. + String knownBugTestEnabled = context.config.otherConfigs.get("enableIcebergKnownBugTest") + if (knownBugTestEnabled == null || !knownBugTestEnabled.equalsIgnoreCase("true")) { + logger.info("skip isolated Iceberg known-bug regression") + return + } + + String restPort = context.config.otherConfigs.get("iceberg_rest_uri_port") + String minioPort = context.config.otherConfigs.get("iceberg_minio_port") + String externalEnvIp = context.config.otherConfigs.get("externalEnvIp") + String catalogName = "test_iceberg_write_required_null_select_negative" + String dbName = "iceberg_write_required_null_select_negative_db" + String internalDb = "iceberg_write_required_null_select_negative_internal_db" + + sql """drop database if exists internal.${internalDb} force""" + sql """create database internal.${internalDb}""" + sql """ + create table internal.${internalDb}.nullable_source ( + id int, + required_text string + ) + duplicate key(id) + distributed by hash(id) buckets 3 + properties ("replication_num" = "1") + """ + sql """ + insert into internal.${internalDb}.nullable_source values + (1, 'valid'), + (2, null) + """ + + sql """drop catalog if exists ${catalogName}""" + sql """ + create catalog ${catalogName} properties ( + "type" = "iceberg", + "iceberg.catalog.type" = "rest", + "uri" = "http://${externalEnvIp}:${restPort}", + "s3.access_key" = "admin", + "s3.secret_key" = "password", + "s3.endpoint" = "http://${externalEnvIp}:${minioPort}", + "s3.region" = "us-east-1" + ) + """ + sql """switch ${catalogName}""" + sql """drop database if exists ${dbName} force""" + sql """create database ${dbName}""" + sql """use ${dbName}""" + + sql """ + create table required_select ( + id int not null, + required_text string not null + ) + partition by list (bucket(8, id)) () + properties ("format-version" = "2") + """ + + long snapshotsBefore = (sql """select count(*) from required_select\$snapshots""")[0][0] as long + long filesBefore = (sql """select count(*) from required_select\$files""")[0][0] as long + long rowsBefore = (sql """select count(*) from required_select""")[0][0] as long + + // W07-S02: The metadata and row baselines distinguish an early rejection + // from a late failure that has already published a snapshot or data file. + test { + sql """ + insert into required_select + select id, required_text + from internal.${internalDb}.nullable_source + """ + exception "null" + } + assertEquals(snapshotsBefore, + (sql """select count(*) from required_select\$snapshots""")[0][0] as long) + assertEquals(filesBefore, + (sql """select count(*) from required_select\$files""")[0][0] as long) + assertEquals(rowsBefore, + (sql """select count(*) from required_select""")[0][0] as long) +} diff --git a/regression-test/suites/external_table_p0/iceberg/write/test_iceberg_write_required_null_values_negative.groovy b/regression-test/suites/external_table_p0/iceberg/write/test_iceberg_write_required_null_values_negative.groovy new file mode 100644 index 00000000000000..86c433dc634554 --- /dev/null +++ b/regression-test/suites/external_table_p0/iceberg/write/test_iceberg_write_required_null_values_negative.groovy @@ -0,0 +1,81 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +suite("test_iceberg_write_required_null_values_negative", + "p0,external,iceberg,external_docker,external_docker_iceberg") { + String enabled = context.config.otherConfigs.get("enableIcebergTest") + if (enabled == null || !enabled.equalsIgnoreCase("true")) { + logger.info("disable iceberg test") + return + } + + // This opt-in switch isolates a write that can publish an unreadable Iceberg data file. + String knownBugTestEnabled = context.config.otherConfigs.get("enableIcebergKnownBugTest") + if (knownBugTestEnabled == null || !knownBugTestEnabled.equalsIgnoreCase("true")) { + logger.info("skip isolated Iceberg known-bug regression") + return + } + + String restPort = context.config.otherConfigs.get("iceberg_rest_uri_port") + String minioPort = context.config.otherConfigs.get("iceberg_minio_port") + String externalEnvIp = context.config.otherConfigs.get("externalEnvIp") + String catalogName = "test_iceberg_write_required_null_values_negative" + String dbName = "iceberg_write_required_null_values_negative_db" + + sql """drop catalog if exists ${catalogName}""" + sql """ + create catalog ${catalogName} properties ( + "type" = "iceberg", + "iceberg.catalog.type" = "rest", + "uri" = "http://${externalEnvIp}:${restPort}", + "s3.access_key" = "admin", + "s3.secret_key" = "password", + "s3.endpoint" = "http://${externalEnvIp}:${minioPort}", + "s3.region" = "us-east-1" + ) + """ + sql """switch ${catalogName}""" + sql """drop database if exists ${dbName} force""" + sql """create database ${dbName}""" + sql """use ${dbName}""" + + sql """ + create table required_values ( + id int not null, + required_text string not null + ) + partition by list (bucket(8, id)) () + properties ("format-version" = "2") + """ + + long snapshotsBefore = (sql """select count(*) from required_values\$snapshots""")[0][0] as long + long filesBefore = (sql """select count(*) from required_values\$files""")[0][0] as long + long rowsBefore = (sql """select count(*) from required_values""")[0][0] as long + + // W07-S01: A thrown NULL error is insufficient proof of atomicity; all + // visible and metadata state must remain at the pre-statement baseline. + test { + sql """insert into required_values values (1, null)""" + exception "null" + } + assertEquals(snapshotsBefore, + (sql """select count(*) from required_values\$snapshots""")[0][0] as long) + assertEquals(filesBefore, + (sql """select count(*) from required_values\$files""")[0][0] as long) + assertEquals(rowsBefore, + (sql """select count(*) from required_values""")[0][0] as long) +} diff --git a/regression-test/suites/external_table_p0/iceberg/write/test_iceberg_write_source_models.groovy b/regression-test/suites/external_table_p0/iceberg/write/test_iceberg_write_source_models.groovy new file mode 100644 index 00000000000000..c98e270ddd3913 --- /dev/null +++ b/regression-test/suites/external_table_p0/iceberg/write/test_iceberg_write_source_models.groovy @@ -0,0 +1,220 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +suite("test_iceberg_write_source_models", + "p0,external,iceberg,external_docker,external_docker_iceberg") { + String enabled = context.config.otherConfigs.get("enableIcebergTest") + if (enabled == null || !enabled.equalsIgnoreCase("true")) { + logger.info("disable iceberg test") + return + } + + String restPort = context.config.otherConfigs.get("iceberg_rest_uri_port") + String minioPort = context.config.otherConfigs.get("iceberg_minio_port") + String externalEnvIp = context.config.otherConfigs.get("externalEnvIp") + String catalogName = "test_iceberg_write_source_models" + String dbName = "iceberg_write_source_models_db" + String internalDb = "iceberg_write_source_models_internal_db" + + sql """drop database if exists internal.${internalDb} force""" + sql """create database internal.${internalDb}""" + + // W04-S01: Duplicate model, no source partition, RANDOM distribution with three buckets. + sql """drop table if exists internal.${internalDb}.source_duplicate""" + sql """ + create table internal.${internalDb}.source_duplicate ( + id int, + category varchar(20), + amount bigint + ) + duplicate key(id) + distributed by random buckets 3 + properties ("replication_num" = "1") + """ + sql """ + insert into internal.${internalDb}.source_duplicate values + (1, 'A', 10), + (1, 'A', 11), + (2, null, 20) + """ + + // W04-S02: Unique MOW model, LIST source partition and HASH AUTO buckets. + sql """drop table if exists internal.${internalDb}.source_unique_mow""" + sql """ + create table internal.${internalDb}.source_unique_mow ( + id int, + category varchar(20), + amount bigint + ) + unique key(id, category) + partition by list(category) ( + partition p_ab values in ('A', 'B'), + partition p_null values in (null) + ) + distributed by hash(id) buckets auto + properties ( + "replication_num" = "1", + "enable_unique_key_merge_on_write" = "true" + ) + """ + sql """insert into internal.${internalDb}.source_unique_mow values (10, 'A', 100), (11, null, 110)""" + sql """insert into internal.${internalDb}.source_unique_mow values (10, 'A', 101)""" + + // W04-S03: Unique MOR model, RANGE source partition and fixed HASH buckets. + sql """drop table if exists internal.${internalDb}.source_unique_mor""" + sql """ + create table internal.${internalDb}.source_unique_mor ( + id int, + category varchar(20), + amount bigint + ) + unique key(id) + partition by range(id) ( + partition p_lt_20 values less than (20), + partition p_max values less than maxvalue + ) + distributed by hash(id) buckets 2 + properties ( + "replication_num" = "1", + "enable_unique_key_merge_on_write" = "false" + ) + """ + sql """insert into internal.${internalDb}.source_unique_mor values (20, 'C', 200), (21, 'D', 210)""" + sql """insert into internal.${internalDb}.source_unique_mor values (20, 'C', 201)""" + + // W04-S04: Aggregate model, RANGE source partition and four fixed HASH buckets. + sql """drop table if exists internal.${internalDb}.source_aggregate""" + sql """ + create table internal.${internalDb}.source_aggregate ( + id int, + category varchar(20), + amount bigint sum + ) + aggregate key(id, category) + partition by range(id) ( + partition p_lt_40 values less than (40), + partition p_max values less than maxvalue + ) + distributed by hash(id, category) buckets 4 + properties ("replication_num" = "1") + """ + sql """ + insert into internal.${internalDb}.source_aggregate values + (30, 'E', 300), + (30, 'E', 3), + (31, 'F', 310) + """ + + order_qt_internal_model_oracle """ + select 'duplicate', id, category, amount + from internal.${internalDb}.source_duplicate + union all + select 'unique_mow', id, category, amount + from internal.${internalDb}.source_unique_mow + union all + select 'unique_mor', id, category, amount + from internal.${internalDb}.source_unique_mor + union all + select 'aggregate', id, category, amount + from internal.${internalDb}.source_aggregate + order by 1, 2, 3, 4 + """ + + sql """drop catalog if exists ${catalogName}""" + sql """ + create catalog ${catalogName} properties ( + "type" = "iceberg", + "iceberg.catalog.type" = "rest", + "uri" = "http://${externalEnvIp}:${restPort}", + "s3.access_key" = "admin", + "s3.secret_key" = "password", + "s3.endpoint" = "http://${externalEnvIp}:${minioPort}", + "s3.region" = "us-east-1", + "meta.cache.iceberg.table.ttl-second" = "0", + "meta.cache.iceberg.schema.ttl-second" = "0" + ) + """ + sql """switch ${catalogName}""" + sql """drop database if exists ${dbName} force""" + sql """create database ${dbName}""" + sql """use ${dbName}""" + sql """set enable_fallback_to_original_planner = false""" + + sql """drop table if exists source_model_sink""" + sql """ + create table source_model_sink ( + source_model string not null, + id int, + category string, + amount bigint + ) + partition by list (source_model, bucket(4, category)) () + properties ( + "format-version" = "2", + "write.format.default" = "parquet" + ) + """ + + // W04-S05: Independent INSERT SELECT statements keep each source model's read semantics. + // Multiple source buckets exercise distributed sink writers on more than one BE. + sql """ + insert into source_model_sink + select 'duplicate', id, category, amount + from internal.${internalDb}.source_duplicate + """ + sql """ + insert into source_model_sink + select 'unique_mow', id, category, amount + from internal.${internalDb}.source_unique_mow + """ + sql """ + insert into source_model_sink + select 'unique_mor', id, category, amount + from internal.${internalDb}.source_unique_mor + """ + sql """ + insert into source_model_sink + select 'aggregate', id, category, amount + from internal.${internalDb}.source_aggregate + """ + + order_qt_source_model_sink """ + select source_model, id, category, amount + from source_model_sink + order by source_model, id, category, amount + """ + order_qt_source_model_partition_stats """ + select spec_id, sum(record_count) + from source_model_sink\$partitions + group by spec_id + order by spec_id + """ + + sql """refresh table ${dbName}.source_model_sink""" + spark_iceberg """refresh table demo.${dbName}.source_model_sink""" + def sparkRows = spark_iceberg """ + select source_model, id, category, amount + from demo.${dbName}.source_model_sink + order by source_model, id, category, amount + """ + def dorisRows = sql """ + select source_model, id, category, amount + from source_model_sink + order by source_model, id, category, amount + """ + assertSparkDorisResultEquals(sparkRows, dorisRows) +} diff --git a/regression-test/suites/external_table_p0/iceberg/write/test_iceberg_write_string_transform_metadata.groovy b/regression-test/suites/external_table_p0/iceberg/write/test_iceberg_write_string_transform_metadata.groovy new file mode 100644 index 00000000000000..ea3b8995ec0017 --- /dev/null +++ b/regression-test/suites/external_table_p0/iceberg/write/test_iceberg_write_string_transform_metadata.groovy @@ -0,0 +1,212 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +suite("test_iceberg_write_string_transform_metadata", + "p0,external,iceberg,external_docker,external_docker_iceberg") { + String enabled = context.config.otherConfigs.get("enableIcebergTest") + if (enabled == null || !enabled.equalsIgnoreCase("true")) { + logger.info("disable iceberg test") + return + } + + String restPort = context.config.otherConfigs.get("iceberg_rest_uri_port") + String minioPort = context.config.otherConfigs.get("iceberg_minio_port") + String externalEnvIp = context.config.otherConfigs.get("externalEnvIp") + String catalogName = "test_iceberg_write_string_transform_metadata" + String dbName = "iceberg_write_string_transform_metadata_db" + + sql """drop catalog if exists ${catalogName}""" + sql """ + create catalog ${catalogName} properties ( + "type" = "iceberg", + "iceberg.catalog.type" = "rest", + "uri" = "http://${externalEnvIp}:${restPort}", + "s3.access_key" = "admin", + "s3.secret_key" = "password", + "s3.endpoint" = "http://${externalEnvIp}:${minioPort}", + "s3.region" = "us-east-1", + "meta.cache.iceberg.table.ttl-second" = "0", + "meta.cache.iceberg.schema.ttl-second" = "0" + ) + """ + sql """switch ${catalogName}""" + sql """drop database if exists ${dbName} force""" + sql """create database ${dbName}""" + sql """use ${dbName}""" + + sql """drop table if exists string_transform_metadata""" + sql """ + create table string_transform_metadata ( + id int not null, + p_identity string, + p_bucket string, + p_truncate string not null, + payload string + ) + partition by list (p_identity) () + properties ( + "format-version" = "2", + "write.format.default" = "parquet" + ) + """ + sql """ + alter table string_transform_metadata + add partition key bucket(8, p_bucket) as p_bucket_8 + """ + sql """ + alter table string_transform_metadata + add partition key truncate(2, p_truncate) as p_truncate_2 + """ + + // WS01-S01: Validate the physical partition values, not only logical row + // equality. STRING bucket accepts NULL and truncate preserves valid UTF-8. + sql """ + insert into string_transform_metadata values + (1, 'ascii', 'bucket-a', 'alphabet', 'ascii'), + (2, '中文', '桶-中文', '中文甲', 'cjk'), + (3, 'emoji', '😀-bucket', '😀甲乙', 'emoji'), + (4, concat('e', unhex('CC81')), 'combining-bucket', + concat('e', unhex('CC81'), 'x'), 'combining'), + (5, '', '', '', 'empty'), + (6, null, null, 'null-bucket', 'nullable-bucket') + """ + order_qt_string_transform_rows """ + select id, hex(p_identity), hex(p_bucket), hex(p_truncate), payload + from string_transform_metadata + order by id + """ + order_qt_string_transform_physical_partitions """ + select struct_element(`partition`, 'p_identity') as p_identity_partition, + struct_element(`partition`, 'p_bucket_8') as p_bucket_partition, + hex(struct_element(`partition`, 'p_truncate_2')) as p_truncate_partition, + record_count + from string_transform_metadata\$partitions + order by p_identity_partition, p_bucket_partition, p_truncate_partition + """ + + spark_iceberg """refresh table demo.${dbName}.string_transform_metadata""" + def sparkRows = spark_iceberg """ + select id, p_identity, p_bucket, p_truncate, payload + from demo.${dbName}.string_transform_metadata + order by id + """ + def dorisRows = sql """ + select id, p_identity, p_bucket, p_truncate, payload + from string_transform_metadata + order by id + """ + assertSparkDorisResultEquals(sparkRows, dorisRows) + + def sparkPartitions = spark_iceberg """ + select partition.p_identity, + partition.p_bucket_8, + hex(partition.p_truncate_2), + record_count + from demo.${dbName}.string_transform_metadata.partitions + order by partition.p_identity, partition.p_bucket_8, hex(partition.p_truncate_2) + """ + def dorisPartitions = sql """ + select struct_element(`partition`, 'p_identity'), + struct_element(`partition`, 'p_bucket_8'), + hex(struct_element(`partition`, 'p_truncate_2')), + record_count + from string_transform_metadata\$partitions + order by struct_element(`partition`, 'p_identity'), + struct_element(`partition`, 'p_bucket_8'), + hex(struct_element(`partition`, 'p_truncate_2')) + """ + assertSparkDorisResultEquals(sparkPartitions, dorisPartitions) + + // WS01-S02: Evolve the bucket and truncate widths and verify that both + // physical specs remain readable by Doris and Spark. + sql """ + alter table string_transform_metadata + replace partition key p_bucket_8 with bucket(16, p_bucket) as p_bucket_16 + """ + sql """ + alter table string_transform_metadata + replace partition key p_truncate_2 with truncate(3, p_truncate) as p_truncate_3 + """ + sql """ + insert into string_transform_metadata values + (7, 'new', 'bucket-new', '中文甲乙', 'new-cjk'), + (8, null, null, '😀甲乙丙', 'new-null-bucket') + """ + order_qt_string_transform_evolved_specs """ + select spec_id, count(*), sum(record_count) + from string_transform_metadata\$partitions + group by spec_id + order by spec_id + """ + // Old and new transform fields must coexist in the union partition struct; + // spec counts alone cannot distinguish bucket(8) from bucket(16), or + // truncate(2) from truncate(3). + order_qt_string_transform_evolved_physical_partitions """ + select spec_id, + hex(struct_element(`partition`, 'p_identity')), + struct_element(`partition`, 'p_bucket_8'), + hex(struct_element(`partition`, 'p_truncate_2')), + struct_element(`partition`, 'p_bucket_16'), + hex(struct_element(`partition`, 'p_truncate_3')), + record_count + from string_transform_metadata\$partitions + order by spec_id, 2, 3, 4, 5, 6 + """ + order_qt_string_transform_evolved_rows """ + select id, hex(p_identity), hex(p_bucket), hex(p_truncate), payload + from string_transform_metadata + order by id + """ + spark_iceberg """refresh table demo.${dbName}.string_transform_metadata""" + sparkRows = spark_iceberg """ + select id, p_identity, p_bucket, p_truncate, payload + from demo.${dbName}.string_transform_metadata + order by id + """ + dorisRows = sql """ + select id, p_identity, p_bucket, p_truncate, payload + from string_transform_metadata + order by id + """ + assertSparkDorisResultEquals(sparkRows, dorisRows) + + // Compare the evolved physical metadata as well as logical rows, including + // both superseded and current transform fields. + sparkPartitions = spark_iceberg """ + select spec_id, + hex(partition.p_identity), + partition.p_bucket_8, + hex(partition.p_truncate_2), + partition.p_bucket_16, + hex(partition.p_truncate_3), + record_count + from demo.${dbName}.string_transform_metadata.partitions + order by spec_id, 2, 3, 4, 5, 6 + """ + dorisPartitions = sql """ + select spec_id, + hex(struct_element(`partition`, 'p_identity')), + struct_element(`partition`, 'p_bucket_8'), + hex(struct_element(`partition`, 'p_truncate_2')), + struct_element(`partition`, 'p_bucket_16'), + hex(struct_element(`partition`, 'p_truncate_3')), + record_count + from string_transform_metadata\$partitions + order by spec_id, 2, 3, 4, 5, 6 + """ + assertSparkDorisResultEquals(sparkPartitions, dorisPartitions) +} diff --git a/regression-test/suites/external_table_p0/iceberg_paimon_schema_time_travel_coverage.md b/regression-test/suites/external_table_p0/iceberg_paimon_schema_time_travel_coverage.md new file mode 100644 index 00000000000000..3f69a2cce356f4 --- /dev/null +++ b/regression-test/suites/external_table_p0/iceberg_paimon_schema_time_travel_coverage.md @@ -0,0 +1,180 @@ +# Iceberg / Paimon Schema Evolution × Time Travel P0 Coverage + +## Scope + +This document maps the P0 regression suites that validate Doris reads after +Iceberg or Paimon schema evolution. The matrix intentionally combines schema +changes with snapshot, tag, branch, time travel, delete/upsert and reader +variants instead of testing these dimensions in isolation. + +The tests use explicit old/new field projections and negative bindings in +addition to row-shape assertions. This prevents a rename with unchanged values +from passing accidentally. + +The partition-evolution extension also distinguishes result correctness from +physical pruning. Static predicates are validated across files written by +different partition specs. Runtime filters on identity partition fields require +equal results with pruning disabled/enabled and a positive pruning counter; +bucket-source and temporal-transform joins assert result parity because those +transform paths are not physically runtime-filter-prunable. + +## Partition evolution matrix + +### Iceberg + +| Partition operation / transform | Static filter | Runtime filter | Snapshot/tag/branch | Position delete | Equality delete | Deletion vector | Complex schema in same timeline | Format / reader | +| --- | --- | --- | --- | --- | --- | --- | --- | --- | +| Add identity field | Covered | Covered | Covered | Covered | Covered | Covered | Covered | Parquet/ORC, V1/V2 | +| Drop identity field | Covered | Covered | Covered | Covered | Covered | Covered | Covered | Parquet/ORC, V1/V2 | +| Add/drop bucket | Covered | Result contract; source transform is not RF-prunable | Covered | Covered | Covered | Covered | Covered | Parquet/ORC, V1/V2 | +| Replace bucket with truncate | Covered | Result contract; transform is not RF-prunable | Covered | Covered by delete timeline | Covered by delete timeline | Covered by delete timeline | Covered | Parquet/ORC, V1/V2 | +| Year → month → day → hour | Covered by one current predicate spanning all four specs | Result contract; temporal transforms are not RF-prunable | Covered | Day → month covered | Day → month covered | Day → month covered | Covered | Parquet/ORC, V1/V2 | +| Old spec lacks newly added field | Covered | Covered | Covered | Covered | Covered | Covered | Covered | Parquet/ORC, V1/V2 | +| New spec lacks dropped field | Covered | Covered | Covered | Covered | Covered | Covered | Covered | Parquet/ORC, V1/V2 | +| Partition-field alias replacement | Covered by existing position-delete suite | Not RF-prunable separately | Existing metadata coverage | Covered | N/A | N/A | N/A | Parquet | + +The Iceberg delete timelines write deletes both before and after spec changes. +Queries then combine partition predicates, runtime filters and historical +references so deleted rows cannot be resurrected or over-deleted when Doris +plans files from several specs. + +### Paimon + +Paimon does not support changing the partition-key set after table creation. +Its P0 contract therefore combines supported schema reordering and payload +evolution with fixed partition keys, and verifies that unsupported key +mutations fail atomically. + +| Paimon partition contract | Static filter | Runtime filter | Snapshot/tag/branch | Delete/upsert/DV | Complex schema | Format / reader | +| --- | --- | --- | --- | --- | --- | --- | +| Reorder partition columns in schema | Covered | N/A | Covered by pre-reorder tag | N/A | Add payload field | Parquet | +| Rename partition key | Rejected atomically | N/A | Historical tag unchanged | Data unchanged | Schema count unchanged | Parquet | +| Change partition-key type | Rejected atomically | N/A | Historical tag unchanged | Data unchanged | Schema count unchanged | Parquet | +| Drop partition key | Rejected atomically | N/A | Historical tag unchanged | Data unchanged | Schema count unchanged | Parquet | +| Fixed key + STRUCT/MAP/ARRAY evolution | Covered | Covered with positive pruning counter | Covered | Append timeline | Current and historical nested projections plus wrong-schema negatives | Parquet/ORC; JNI and native split paths asserted | +| Partitioned PK table | Covered | Covered with positive pruning counter | Covered | Upsert/delete/compaction plus physical DV artifact | Current and historical evolved fields, including drop/re-add NULL isolation | Parquet/ORC; JNI and native split paths asserted | + +## Schema operations + +| ID | Operation | Iceberg | Paimon | P0 contract | +| --- | --- | --- | --- | --- | +| S01 | Add one nullable top-level field | Positive | Positive | Old refs reject the new field; new refs backfill NULL | +| S02 | Add multiple top-level fields | Positive | Positive | Order, types and NULL backfill are snapshot-local | +| S03 | Reorder / FIRST / AFTER | Positive | Positive | Field IDs remain stable across old and new refs | +| S04 | Rename top-level field | Positive | Positive | Old and new names bind only in their own schemas | +| S05 | Case-only or mixed-case rename | Positive | Positive | Case handling does not hide schema selection errors | +| S06 | Drop top-level field | Positive | Positive | Old refs retain the field; current refs reject it | +| S07 | Drop and re-add the same name | Positive | Positive | Old and new field IDs never share values | +| S08 | Compatible type promotion | Positive | Positive | Historical and current types remain correct | +| S09 | Add STRUCT child | Positive | Positive | Nested field is visible only after its schema version | +| S10 | Rename STRUCT child | Positive | Positive | Nested old/new paths have snapshot-local binding | +| S11 | Drop STRUCT child | Positive | Positive | Historical nested projections remain readable | +| S12 | Drop and re-add STRUCT child | Positive | Positive | Nested field IDs never leak values | +| S13 | Promote/reorder STRUCT child | Positive | Positive | Nested predicate and projection use the right type | +| S14 | MAP value STRUCT evolution | Positive | Positive | `element_at(...).field` uses the selected schema | +| S15 | ARRAY element STRUCT evolution | Positive | Positive | `array[i].field` uses the selected schema | +| S16 | Partition-column mutation | Positive/restricted | Negative format contract | Unsupported mutations are rejected atomically | +| S17 | Partition evolution | Positive | Format-constrained contract | Data and payload evolution preserve partition reads | +| S18 | PK-table non-key evolution | N/A | Positive | Upsert/delete/DV remain correct through evolution | +| S19 | Comment/default/nullability | Positive and negative | Positive and negative | Metadata-only changes and rejections are atomic | +| S20 | Narrowing, map-key, key/partition mutations | Negative | Negative | No schema, snapshot or cached state is polluted | + +## Historical references and actions + +| ID | Reference/action | Iceberg | Paimon | Coverage | +| --- | --- | --- | --- | --- | +| T00 | Latest/current | Yes | Yes | Current schema, explicit projection, predicate, aggregate | +| T01/T02 | Pre/post numeric snapshot | Yes | Yes | Old/new field binding and row visibility | +| T03 | Timestamp string | Yes | Yes | Resolves the expected pre-change schema | +| T04 | Epoch millis | Stable rejection | Yes | Iceberg syntax rejection; Paimon positive read | +| T05/T06/T07 | Pre/post tag forms | Yes | Yes | `FOR VERSION AS OF` and `@tag` | +| T08 | Pre-change branch | Negative product contract | Yes | Branch schema/data are compared with the format oracle | +| T09 | Independent branch evolution | Negative product contract | Negative product contract | Failure is isolated without crashing the shared BE | +| T10 | Expired snapshot retained by tag | Existing lifecycle + matrix | Yes | Retained tag never falls back to latest | +| T11 | Missing snapshot/tag | Yes | Yes | Stable error, never latest fallback | +| T12/T13 | Dual historical relations | Negative product contract | Negative product contract | Join, reverse join, UNION, nested UNION, CTE, subquery | +| T14 | Incremental read across change | N/A | Yes | JNI/CPP-supported paths use the end schema | +| T15 | Rollback to snapshot/timestamp | Yes | N/A | Data rolls back while Iceberg current schema semantics remain correct | +| T16 | Cherry-pick / fast-forward | Yes | Paimon fast-forward oracle | Current, tag and branch state are verified | + +## Delete and execution dimensions + +| Dimension | Iceberg | Paimon | +| --- | --- | --- | +| Position delete | v2, Parquet and ORC, before/after evolution | N/A | +| Equality delete | Rename, promotion, drop/re-add, old/new snapshots | N/A | +| Deletion vector | v3, Parquet and ORC, before/after evolution | PK-table DV path | +| Row operations | Delete visibility around every checkpoint | Upsert, delete and compaction | +| Readers | Complete identity/bucket/truncate/drop and year/month/day/hour timelines under file scanner V1/V2, for both Parquet and ORC | JNI and native paths are proved by split statistics for current and historical projections | +| Cache | REST cache on/off | Filesystem metadata cache on/off | +| Catalog smoke | REST full matrix and JDBC rename/time-travel | Filesystem full matrix and JDBC rename/time-travel | +| Cluster topology | External endpoints are cluster-reachable; JDBC drivers are installed on every FE/BE | External endpoints are cluster-reachable; JDBC drivers are installed on every FE/BE | + +HMS and DLF are credential/environment variants rather than additional schema +semantics. Their existing catalog suites remain responsible for connectivity; +the deterministic P0 schema matrix uses REST/filesystem, and JDBC gets an +explicit rename plus old snapshot/tag smoke path. + +## Suite map + +| Suite | Responsibility | +| --- | --- | +| `iceberg/test_iceberg_schema_time_travel_matrix.groovy` | S01-S17 × T00-T13, nested evolution, partition evolution, cache/scanner variants | +| `iceberg/test_iceberg_schema_dual_relation_matrix.groovy` | Dual-snapshot join/UNION/CTE/subquery negative contracts | +| `iceberg/test_iceberg_schema_equality_delete_time_travel.groovy` | Equality delete × rename/promotion/drop-re-add | +| `iceberg/test_iceberg_schema_position_dv_time_travel.groovy` | Position delete and DV × top-level/nested evolution, Parquet/ORC | +| `iceberg/test_iceberg_schema_ref_actions_matrix.groovy` | Rollback, cherry-pick, fast-forward and branch action semantics | +| `iceberg/test_iceberg_schema_metadata_atomicity_matrix.groovy` | Comment/default/nullability/narrowing/map-key atomicity | +| `iceberg/test_iceberg_partition_evolution_filter_refs.groovy` | Add/drop/replace specs, all transform families, static filters and snapshot/tag/branch | +| `iceberg/test_iceberg_partition_evolution_runtime_filter.groovy` | Runtime-filter pruning across specs with missing/new partition fields | +| `iceberg/test_iceberg_partition_evolution_position_dv.groovy` | Position delete and deletion vector before/after spec changes, filters and refs | +| `iceberg/test_iceberg_partition_evolution_equality_delete.groovy` | Real equality-delete files before/after add/replace/drop specs | +| `iceberg/test_iceberg_partition_evolution_format_scanner.groovy` | Full partition-transform timeline for Parquet/ORC under scanner V1/V2, current and tagged refs | +| `paimon/test_paimon_schema_time_travel_matrix.groovy` | S01-S18 × T00-T14, PK upsert/delete/DV, cache/readers | +| `paimon/test_paimon_schema_dual_relation_matrix.groovy` | Dual-snapshot join/UNION/CTE/subquery negative contracts | +| `paimon/test_paimon_schema_branch_partition_matrix.groovy` | Independent branch evolution, fast-forward and partition restrictions | +| `paimon/test_paimon_schema_metadata_atomicity_matrix.groovy` | Comment/default/nullability/narrowing atomicity | +| `paimon/test_paimon_partition_schema_filter_refs.groovy` | Fixed partition key with complex evolution, pruning and historical refs | +| `paimon/test_paimon_partition_pk_delete_refs.groovy` | Partitioned PK upsert/delete/DV/compaction with filters and refs | +| `paimon/test_paimon_partition_mutation_atomicity.groovy` | Supported reorder plus atomic rejection of key rename/type/drop | +| `iceberg/test_iceberg_jdbc_catalog.groovy` | JDBC catalog rename × numeric snapshot/tag smoke | +| `paimon/test_paimon_jdbc_catalog.groovy` | JDBC catalog rename × numeric snapshot/tag smoke | + +Each Groovy file contains `Scenario` comments identifying the matrix cell under +test. Internal issue identifiers are deliberately absent from the checked-in +test sources and coverage documentation. + +## Product contracts discovered by the matrix + +| Contract | Negative regression location | +| --- | --- | +| Iceberg historical branch schema isolation | `test_iceberg_schema_time_travel_matrix.groovy`, `test_iceberg_schema_ref_actions_matrix.groovy` | +| Iceberg dual-relation schema isolation | `test_iceberg_schema_dual_relation_matrix.groovy` | +| Paimon dual-relation schema isolation | `test_paimon_schema_dual_relation_matrix.groovy` | +| Paimon branch schema initialization isolation | `test_paimon_schema_branch_partition_matrix.groovy` | +| Iceberg nested field comment preservation | `test_iceberg_schema_metadata_atomicity_matrix.groovy` | + +## Validation status + +- All eight partition-evolution extension suites pass against master + `f1460f89230441bc1b6b1872d66bd32a526b25b1`: no failed, fatal or skipped + suite. +- The matrix contains twenty P0 suites: ten original REST/filesystem suites, + two JDBC catalog suites and eight partition-evolution extension suites. +- Independent REST/filesystem suites support parallel execution. The four + profile-dependent partition extensions are marked `nonConcurrent` so profile + lookup and session-counter assertions cannot consume another suite's query; + the two existing JDBC catalog suites are validated separately because they + initialize shared external catalog fixtures. +- Distributed validation used one FE and two live BEs. The cases use + cluster-reachable external endpoints and contain no fixed backend, + backend-local path or single-node scheduling assumption. +- No additional Doris product issue was reproduced by the partition-evolution + extension. Test-fixture and Runtime Filter session preconditions found during + development were corrected before the final run and were not recorded as + product defects. + +The requested schema-change × partition-operation × historical-reference +matrix has no unimplemented P0 cell. Unsupported format operations and +currently incorrect Doris behavior are represented by stable negative +regression contracts rather than being marked as missing. diff --git a/regression-test/suites/external_table_p0/paimon/test_paimon_jdbc_catalog.groovy b/regression-test/suites/external_table_p0/paimon/test_paimon_jdbc_catalog.groovy index b5f233aac67515..cbb3174ea5faeb 100644 --- a/regression-test/suites/external_table_p0/paimon/test_paimon_jdbc_catalog.groovy +++ b/regression-test/suites/external_table_p0/paimon/test_paimon_jdbc_catalog.groovy @@ -63,6 +63,8 @@ suite("test_paimon_jdbc_catalog", "p0,external") { String localDriverPath = "${localDriverDir}/${driverName}" String sparkDriverPath = "/tmp/${driverName}" String sparkSeedCatalogName = "${catalogName}_seed" + // Reuse the fixture-wide Docker command so local and CI permission models behave identically. + String dockerCommand = context.config.otherConfigs.get("externalDockerCommand") ?: "docker" assertTrue(jdbcDriversDir != null && !jdbcDriversDir.isEmpty(), "jdbc_drivers_dir must be configured") @@ -90,25 +92,63 @@ suite("test_paimon_jdbc_catalog", "p0,external") { } executeCommand("mkdir -p ${localDriverDir}", false, 60) - executeCommand("mkdir -p ${jdbcDriversDir}", true, 60) if (!new File(localDriverPath).exists()) { executeCommand("/usr/bin/curl --max-time 600 ${driverDownloadUrl} --output ${localDriverPath}", true, 660) } - executeCommand("cp -f ${localDriverPath} ${jdbcDriversDir}/${driverName}", true, 60) - String sparkContainerName = executeCommand("docker ps --filter name=spark-iceberg --format {{.Names}}", false, 30) + def clusterHostIps = new ArrayList() + String[][] backends = sql """show backends""" + for (def backend in backends) { + clusterHostIps.add(backend[1]) + } + String[][] frontends = sql """show frontends""" + for (def frontend in frontends) { + clusterHostIps.add(frontend[1]) + } + clusterHostIps = clusterHostIps.unique() + + Set localHostIps = ["127.0.0.1", "localhost", "::1"] as Set + java.util.Collections.list(java.net.NetworkInterface.getNetworkInterfaces()).each { networkInterface -> + java.util.Collections.list(networkInterface.getInetAddresses()).each { address -> + localHostIps.add(address.getHostAddress().split("%")[0]) + } + } + localHostIps.add(java.net.InetAddress.getLocalHost().getHostName()) + localHostIps.add(java.net.InetAddress.getLocalHost().getCanonicalHostName()) + + for (def hostIp in clusterHostIps) { + // Scenario: every FE/BE receives the JDBC driver so metadata failover and distributed + // scan scheduling do not depend on a driver installed only on the regression runner. + if (localHostIps.contains(hostIp)) { + executeCommand("mkdir -p ${jdbcDriversDir}", true, 60) + executeCommand("cp -f ${localDriverPath} ${jdbcDriversDir}/${driverName}", true, 60) + } else { + executeCommand( + "ssh -o BatchMode=yes -o StrictHostKeyChecking=no root@${hostIp} \"mkdir -p ${jdbcDriversDir}\"", + true, + 60 + ) + scpFiles("root", hostIp, localDriverPath, jdbcDriversDir, false) + } + } + + String sparkContainerName = executeCommand( + "${dockerCommand} ps --filter name=spark-iceberg --format {{.Names}}", + false, + 30 + ) ?.trim() if (sparkContainerName == null || sparkContainerName.isEmpty()) { logger.info("spark-iceberg container not found, skip this test") return } - executeCommand("docker cp ${localDriverPath} ${sparkContainerName}:${sparkDriverPath}", true, 60) + executeCommand("${dockerCommand} cp ${localDriverPath} ${sparkContainerName}:${sparkDriverPath}", true, 60) String sparkMinioEndpoint = "http://${externalEnvIp}:${minioPort}" if (sparkContainerName.contains("spark-iceberg")) { String sparkMinioContainerName = sparkContainerName.replaceFirst("spark-iceberg", "minio") String resolvedSparkMinioContainer = executeCommand( - "docker ps --filter name=${sparkMinioContainerName} --format {{.Names}}", + "${dockerCommand} ps --filter name=${sparkMinioContainerName} --format {{.Names}}", false, 30 )?.trim() @@ -121,7 +161,7 @@ suite("test_paimon_jdbc_catalog", "p0,external") { def sparkPaimonJdbc = { String sqlText -> String escapedSql = sqlText.replaceAll('"', '\\\\"') - String command = """docker exec ${sparkContainerName} spark-sql --master spark://${sparkContainerName}:7077 \ + String command = """${dockerCommand} exec ${sparkContainerName} spark-sql --master spark://${sparkContainerName}:7077 \ --jars ${sparkDriverPath} \ --driver-class-path ${sparkDriverPath} \ --conf spark.driver.extraClassPath=${sparkDriverPath} \ @@ -225,6 +265,43 @@ suite("test_paimon_jdbc_catalog", "p0,external") { assertEquals(1, rowCount.size()) assertEquals("2", rowCount[0][0].toString()) + // Scenario TC09-JDBC: Paimon JDBC catalog preserves the old snapshot schema after rename. + String jdbcOldSnapshot = sql(""" + select snapshot_id + from paimon_jdbc_tbl\$snapshots + order by snapshot_id desc + limit 1 + """)[0][0].toString() + sparkPaimonJdbc """ + CALL ${sparkSeedCatalogName}.sys.create_tag( + table => '${dbName}.paimon_jdbc_tbl', + tag => 'jdbc_before_rename', + snapshot => ${jdbcOldSnapshot} + ) + """ + sparkPaimonJdbc """ + ALTER TABLE ${sparkSeedCatalogName}.${dbName}.paimon_jdbc_tbl + RENAME COLUMN name TO jdbc_renamed_name + """ + sql """REFRESH TABLE paimon_jdbc_tbl""" + assertEquals([[1, "alice"], [2, "bob"]], sql(""" + select id, name + from paimon_jdbc_tbl for version as of ${jdbcOldSnapshot} + order by id + """)) + assertEquals([[1, "alice"], [2, "bob"]], sql(""" + select id, name + from paimon_jdbc_tbl@tag(jdbc_before_rename) + order by id + """)) + assertEquals([[1, "alice"], [2, "bob"]], sql(""" + select id, jdbc_renamed_name from paimon_jdbc_tbl order by id + """)) + test { + sql """select name from paimon_jdbc_tbl""" + exception "Unknown column 'name'" + } + assertSystemTableReadable("paimon_jdbc_tbl\$schemas", ["schema_id"], 1) assertSystemTableReadable("paimon_jdbc_tbl\$snapshots", ["snapshot_id"], 1) [ diff --git a/regression-test/suites/external_table_p0/paimon/test_paimon_partition_mutation_atomicity.groovy b/regression-test/suites/external_table_p0/paimon/test_paimon_partition_mutation_atomicity.groovy new file mode 100644 index 00000000000000..e86c5af0e9df30 --- /dev/null +++ b/regression-test/suites/external_table_p0/paimon/test_paimon_partition_mutation_atomicity.groovy @@ -0,0 +1,148 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +suite("test_paimon_partition_mutation_atomicity", + "p0,external,paimon,external_docker,external_docker_paimon") { + String enabled = context.config.otherConfigs.get("enablePaimonTest") + if (enabled == null || !enabled.equalsIgnoreCase("true")) { + logger.info("disable paimon test") + return + } + + String minioPort = context.config.otherConfigs.get("iceberg_minio_port") + String externalEnvIp = context.config.otherConfigs.get("externalEnvIp") + String catalogName = "test_paimon_partition_mutation_atomicity" + String dbName = "paimon_partition_mutation_atomicity_db" + String tableName = "partition_contract" + + def schemaCount = { + return spark_paimon(""" + select count(*) from paimon.${dbName}.`${tableName}\$schemas` + """)[0][0].toString().toInteger() + } + def assertSparkRejected = { String statement, String operation, String expectedMessage -> + String error = null + try { + spark_paimon(statement) + } catch (Exception e) { + error = e.getMessage() + } + assertNotNull(error, "Paimon must reject partition-key ${operation}") + assertTrue(error.contains(expectedMessage), + "Partition-key ${operation} should report '${expectedMessage}': ${error}") + } + + sql """drop catalog if exists ${catalogName}""" + sql """ + create catalog ${catalogName} properties ( + 'type'='paimon', + 'warehouse'='s3://warehouse/wh', + 's3.endpoint'='http://${externalEnvIp}:${minioPort}', + 's3.access_key'='admin', + 's3.secret_key'='password', + 's3.path.style.access'='true', + 'meta.cache.paimon.table.ttl-second'='0' + ) + """ + + try { + spark_paimon_multi """ + create database if not exists paimon.${dbName}; + drop table if exists paimon.${dbName}.${tableName}; + create table paimon.${dbName}.${tableName} ( + id int, + part_a string, + payload string, + part_b int + ) using paimon + partitioned by (part_a, part_b) + tblproperties ('file.format'='parquet'); + insert into paimon.${dbName}.${tableName} values + (1, 'A', 'base-a1', 1), + (2, 'A', 'base-a2', 2), + (3, 'B', 'base-b1', 1); + call paimon.sys.create_tag( + table => '${dbName}.${tableName}', + tag => 'partition_contract_base' + ); + """ + + // Scenario PM-M01-positive: partition columns may move in schema order without changing + // their partition identity, values, tag or filter semantics. + spark_paimon_multi """ + alter table paimon.${dbName}.${tableName} alter column part_b first; + alter table paimon.${dbName}.${tableName} alter column part_a after id; + alter table paimon.${dbName}.${tableName} add column added_payload string after payload; + insert into paimon.${dbName}.${tableName} + (part_b, id, part_a, payload, added_payload) + values (3, 4, 'A', 'after-reorder', 'added'); + """ + int schemasAfterSupportedChanges = schemaCount() + + // Scenario PM-M02-negative: rename, type change and drop of either partition key must be + // rejected atomically. These are format restrictions, not missing Doris capabilities. + assertSparkRejected(""" + alter table paimon.${dbName}.${tableName} + rename column part_a to renamed_part_a + """, "rename", "Cannot rename partition column") + assertSparkRejected(""" + alter table paimon.${dbName}.${tableName} + alter column part_b type bigint + """, "type change", "Cannot update partition column") + assertSparkRejected(""" + alter table paimon.${dbName}.${tableName} drop column part_a + """, "drop", "Cannot drop partition key or primary key") + assertSparkRejected(""" + alter table paimon.${dbName}.${tableName} drop column part_b + """, "drop", "Cannot drop partition key or primary key") + assertEquals(schemasAfterSupportedChanges, schemaCount(), + "Rejected partition mutations must not create Paimon schemas") + + sql """switch ${catalogName}""" + sql """use ${dbName}""" + sql """refresh table ${tableName}""" + + // Scenario PM-M03: current data and static partition filtering survive accepted reorder + // and all rejected mutations. + qt_current_partition_filter """ + select id from ${tableName} + where part_a = 'A' and part_b in (1, 2, 3) + order by id + """ + qt_current_added_payload """ + select id, added_payload from ${tableName} + where part_a = 'A' and part_b = 3 + """ + + // Scenario PM-M04: the pre-reorder tag retains its original schema and partitions. + qt_base_tag_partition_filter """ + select id, part_a, part_b, payload + from ${tableName}@tag(partition_contract_base) + where part_a in ('A', 'B') + order by id + """ + test { + sql """ + select added_payload + from ${tableName}@tag(partition_contract_base) + """ + exception "Unknown column 'added_payload'" + } + } finally { + sql """drop catalog if exists ${catalogName}""" + } +} diff --git a/regression-test/suites/external_table_p0/paimon/test_paimon_partition_pk_delete_refs.groovy b/regression-test/suites/external_table_p0/paimon/test_paimon_partition_pk_delete_refs.groovy new file mode 100644 index 00000000000000..85007d147ba2d0 --- /dev/null +++ b/regression-test/suites/external_table_p0/paimon/test_paimon_partition_pk_delete_refs.groovy @@ -0,0 +1,346 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +import org.apache.doris.regression.action.ProfileAction + +suite("test_paimon_partition_pk_delete_refs", + "p0,external,paimon,external_docker,external_docker_paimon,nonConcurrent") { + String enabled = context.config.otherConfigs.get("enablePaimonTest") + if (enabled == null || !enabled.equalsIgnoreCase("true")) { + logger.info("disable paimon test") + return + } + + String minioPort = context.config.otherConfigs.get("iceberg_minio_port") + String externalEnvIp = context.config.otherConfigs.get("externalEnvIp") + String catalogName = "test_paimon_partition_pk_delete_refs" + String dbName = "paimon_partition_pk_delete_refs_db" + + def stringRows = { String query -> + sql(query).collect { row -> + row.collect { value -> value == null ? null : value.toString() } + } + } + def latestSnapshotId = { String tableName -> + return spark_paimon(""" + select snapshot_id + from paimon.${dbName}.`${tableName}\$snapshots` + order by snapshot_id desc + limit 1 + """)[0][0].toString() + } + def createTag = { String tableName, String tagName -> + spark_paimon """ + call paimon.sys.create_tag( + table => '${dbName}.${tableName}', + tag => '${tagName}' + ) + """ + } + def profileAction = new ProfileAction(context) + def profileCounterValues = { String profileText, String counterName -> + def values = [] + def matcher = profileText =~ ("(?m)^\\s*(?:-\\s*)?" + + java.util.regex.Pattern.quote(counterName) + ":\\s+([^\\n]+)") + while (matcher.find()) { + String valueText = matcher.group(1).toString() + def exact = valueText =~ /\(([0-9,]+)\)/ + def number = valueText =~ /([0-9,]+)/ + String rawValue = exact.find() ? exact.group(1) : (number.find() ? number.group(1) : null) + if (rawValue != null) { + values.add(Long.parseLong(rawValue.replace(",", ""))) + } + } + return values + } + def assertRuntimeFilterPruned = { String tableName, String dimensionTable -> + String token = UUID.randomUUID().toString() + List> rows = stringRows(""" + select /*+ SET_VAR(runtime_filter_type='IN_OR_BLOOM_FILTER') */ + '${token}', f.id, f.full_name + from ${tableName} f + join ${dimensionTable} d on f.part = d.part + order by f.id + """) + String profile = profileAction.getProfileBySql( + token, + ["RuntimeFilterPartitionPrunedRangeNum"], + 30000L, + 500L) + long fileRangesPruned = profileCounterValues( + profile, "RuntimeFilterPartitionPrunedRangeNum").sum(0L) + long partitionsPruned = profileCounterValues( + profile, "PartitionsPrunedByRuntimeFilter").sum(0L) + assertTrue(fileRangesPruned + partitionsPruned > 0L, + "Runtime filter did not prune a Paimon PK partition/file range; " + + profile.take(2000).replaceAll("\\s+", " ")) + return rows.collect { row -> [row[1], row[2]] } + } + def getExplainText = { String query -> + return sql("explain verbose ${query}").collect { row -> row[0].toString() }.join("\n") + } + def assertNativePath = { String query, String label -> + String explainText = getExplainText(query) + def splitMatcher = (explainText =~ /paimonNativeReadSplits=(\d+)\/(\d+)/) + assertTrue(splitMatcher.find(), "Expected paimonNativeReadSplits for ${label}") + long nativeSplits = Long.parseLong(splitMatcher.group(1)) + long totalSplits = Long.parseLong(splitMatcher.group(2)) + assertTrue(totalSplits > 0 && nativeSplits > 0, + "Expected native splits for ${label}, native=${nativeSplits}, total=${totalSplits}") + assertTrue(explainText.contains("SplitStat [type=NATIVE"), + "Expected a NATIVE split for ${label}") + } + def assertJniPath = { String query, String label -> + String explainText = getExplainText(query) + def splitMatcher = (explainText =~ /paimonNativeReadSplits=(\d+)\/(\d+)/) + assertTrue(splitMatcher.find(), "Expected paimonNativeReadSplits for ${label}") + long nativeSplits = Long.parseLong(splitMatcher.group(1)) + long totalSplits = Long.parseLong(splitMatcher.group(2)) + assertTrue(totalSplits > 0 && nativeSplits == 0, + "Expected JNI-only splits for ${label}, native=${nativeSplits}, total=${totalSplits}") + assertTrue(explainText.contains("SplitStat [type=JNI"), + "Expected a JNI split for ${label}") + } + + sql """drop catalog if exists ${catalogName}""" + sql """ + create catalog ${catalogName} properties ( + 'type'='paimon', + 'warehouse'='s3://warehouse/wh', + 's3.endpoint'='http://${externalEnvIp}:${minioPort}', + 's3.access_key'='admin', + 's3.secret_key'='password', + 's3.path.style.access'='true', + 'meta.cache.paimon.table.ttl-second'='0' + ) + """ + + try { + ["parquet", "orc"].each { String format -> + String tableName = "partition_pk_dv_${format}" + String dimensionTable = "${tableName}_dimension" + spark_paimon_multi """ + create database if not exists paimon.${dbName}; + drop table if exists paimon.${dbName}.${tableName}; + create table paimon.${dbName}.${tableName} ( + id int not null, + part string not null, + old_name string, + note string, + payload struct + ) using paimon + partitioned by (part) + tblproperties ( + 'bucket'='1', + 'primary-key'='part,id', + 'file.format'='${format}', + 'deletion-vectors.enabled'='true' + ); + insert into paimon.${dbName}.${tableName} values + (1, 'p1', 'alpha', 'old-note-1', + named_struct('metric', 10, 'label', 'base-1')), + (2, 'p1', 'beta', 'old-note-2', + named_struct('metric', 20, 'label', 'base-delete')), + (3, 'p2', 'gamma', 'old-note-3', + named_struct('metric', 30, 'label', 'base-p2')), + (90, 'p_dv', 'dv-victim', 'old-note-90', + named_struct('metric', 90, 'label', 'dv-victim')); + """ + String baseSnapshot = latestSnapshotId(tableName) + createTag(tableName, "${tableName}_base") + + // Scenario PM-D01: add/rename fields, upsert one PK and delete another inside p1. + spark_paimon_multi """ + alter table paimon.${dbName}.${tableName} add column payload.extra string; + alter table paimon.${dbName}.${tableName} rename column old_name to full_name; + insert into paimon.${dbName}.${tableName} + (id, part, full_name, note, payload) values + (1, 'p1', 'alpha-updated', 'new-note-1', + named_struct('metric', 11, 'label', 'updated-1', 'extra', 'extra-1')), + (4, 'p1', 'delta', 'delete-later', + named_struct('metric', 40, 'label', 'insert-4', 'extra', 'extra-4')); + delete from paimon.${dbName}.${tableName} + where part = 'p1' and id = 2; + """ + String firstDeleteSnapshot = latestSnapshotId(tableName) + createTag(tableName, "${tableName}_first_delete") + + // Scenario PM-D02: nested rename/type promotion and drop/re-add combine with another + // delete, insert and full compaction while the partition key stays fixed. + spark_paimon_multi """ + alter table paimon.${dbName}.${tableName} + rename column payload.label to renamed_label; + alter table paimon.${dbName}.${tableName} + alter column payload.metric type bigint; + alter table paimon.${dbName}.${tableName} drop column note; + alter table paimon.${dbName}.${tableName} add column note bigint; + delete from paimon.${dbName}.${tableName} + where part = 'p1' and id = 4; + insert into paimon.${dbName}.${tableName} + (id, part, full_name, payload, note) values + (5, 'p1', 'epsilon', + named_struct('metric', 5000000000, + 'renamed_label', 'insert-5', 'extra', 'extra-5'), + 5000); + call paimon.sys.compact( + table => '${dbName}.${tableName}', + compact_strategy => 'full' + ); + call paimon.sys.create_tag( + table => '${dbName}.${tableName}', + tag => '${tableName}_pre_dv' + ); + -- Prevent synchronous batch compaction from materializing the sacrificial + -- delete, so the fixture retains a physical DV artifact for reader checks. + alter table paimon.${dbName}.${tableName} + set tblproperties ('write-only'='true'); + delete from paimon.${dbName}.${tableName} + where part = 'p_dv' and id = 90; + drop table if exists paimon.${dbName}.${dimensionTable}; + create table paimon.${dbName}.${dimensionTable} (part string) + using paimon tblproperties ('file.format'='parquet'); + insert into paimon.${dbName}.${dimensionTable} values ('p1'); + """ + long deletedRowsInFiles = spark_paimon(""" + select coalesce(sum(deleteRowCount), 0) + from paimon.${dbName}.`${tableName}\$files` + """)[0][0].toString().toLong() + assertTrue(deletedRowsInFiles > 0, + "${tableName} must retain a physical deletion-vector row") + String finalSnapshot = latestSnapshotId(tableName) + createTag(tableName, "${tableName}_final") + + sql """switch ${catalogName}""" + sql """use ${dbName}""" + sql """refresh table ${tableName}""" + String actionSuffix = format + + // Scenario PM-D03: static partition filters apply PK upserts, deletes and DV state. + List> expectedCurrent = [["1", "alpha-updated"], ["5", "epsilon"]] + "qt_${actionSuffix}_current_partition_filter"(""" + select id, full_name from ${tableName} + where part = 'p1' order by id + """) + "qt_${actionSuffix}_current_promoted_and_readded"(""" + select id, payload.metric, note from ${tableName} + where part = 'p1' and payload.metric > 1000000000 + order by id + """) + // Surviving pre-readd row 1 must see NULL for the new BIGINT note field, while the + // nested rename and added child stay bound by field ID across the delete timeline. + "qt_${actionSuffix}_current_evolved_fields"(""" + select id, payload.renamed_label, payload.extra, note + from ${tableName} + where part = 'p1' + order by id + """) + "qt_${actionSuffix}_pre_dv_old_row_null"(""" + select id, payload.renamed_label, payload.extra, note + from ${tableName}@tag(${tableName}_pre_dv) + where part = 'p1' and id = 1 + """) + test { + sql """select payload.label from ${tableName} where part = 'p1'""" + exception "label" + } + + // Scenario PM-D04: runtime-filter pruning on the partition key remains delete-aware. + String rfQuery = """ + select /*+ SET_VAR(runtime_filter_type='IN_OR_BLOOM_FILTER') */ + f.id, f.full_name + from ${tableName} f + join ${dimensionTable} d on f.part = d.part + order by f.id + """ + sql """set runtime_filter_wait_infinitely=true""" + sql """set disable_join_reorder=true""" + sql """set enable_runtime_filter_prune=false""" + sql """set runtime_filter_mode=GLOBAL""" + sql """set parallel_pipeline_task_num=1""" + sql """set enable_profile=true""" + sql """set profile_level=2""" + sql """set enable_runtime_filter_partition_prune=false""" + "qt_${actionSuffix}_rf_disabled"(rfQuery) + sql """set enable_runtime_filter_partition_prune=true""" + assertEquals(expectedCurrent, assertRuntimeFilterPruned(tableName, dimensionTable)) + + // Scenario PM-D05: numeric snapshots and tags preserve the matching schema/delete set. + "qt_${actionSuffix}_base_snapshot"(""" + select id, old_name, note, payload.label + from ${tableName} for version as of ${baseSnapshot} + where part = 'p1' order by id + """) + "qt_${actionSuffix}_base_tag"(""" + select id, old_name, note, payload.label + from ${tableName}@tag(${tableName}_base) + where part = 'p1' order by id + """) + "qt_${actionSuffix}_first_delete_snapshot"(""" + select id, full_name, note, payload.label, payload.extra + from ${tableName} for version as of ${firstDeleteSnapshot} + where part = 'p1' order by id + """) + "qt_${actionSuffix}_final_tag"(""" + select id, full_name, note, payload.renamed_label, payload.extra + from ${tableName}@tag(${tableName}_final) + where part = 'p1' order by id + """) + test { + sql """ + select payload.renamed_label + from ${tableName} for version as of ${baseSnapshot} + """ + exception "renamed_label" + } + + // Scenario PM-D06: JNI/native readers agree for current DV state and historical + // projections, and explain must prove the requested reader path rather than fallback. + String currentReaderQuery = """ + select id, part, full_name, note, payload.renamed_label, payload.extra + from ${tableName} + where part in ('p1', 'p2') + order by part, id + """ + String historicalReaderQuery = """ + select id, part, old_name, note, payload.label + from ${tableName} for version as of ${baseSnapshot} + where part in ('p1', 'p2') + order by part, id + """ + sql """set enable_paimon_cpp_reader=false""" + sql """set force_jni_scanner=true""" + assertJniPath(currentReaderQuery, "${tableName} current DV") + assertJniPath(historicalReaderQuery, "${tableName} historical") + "qt_${actionSuffix}_jni_current_dv"(currentReaderQuery) + "qt_${actionSuffix}_jni_historical"(historicalReaderQuery) + sql """set force_jni_scanner=false""" + sql """set enable_paimon_cpp_reader=true""" + assertNativePath(currentReaderQuery, "${tableName} current DV") + assertNativePath(historicalReaderQuery, "${tableName} historical") + "qt_${actionSuffix}_native_current_dv"(currentReaderQuery) + "qt_${actionSuffix}_native_historical"(historicalReaderQuery) + assertEquals(finalSnapshot, latestSnapshotId(tableName)) + } + } finally { + sql """set enable_paimon_cpp_reader=false""" + sql """set force_jni_scanner=false""" + sql """set enable_runtime_filter_prune=true""" + sql """set enable_runtime_filter_partition_prune=true""" + sql """set disable_join_reorder=false""" + sql """drop catalog if exists ${catalogName}""" + } +} diff --git a/regression-test/suites/external_table_p0/paimon/test_paimon_partition_schema_filter_refs.groovy b/regression-test/suites/external_table_p0/paimon/test_paimon_partition_schema_filter_refs.groovy new file mode 100644 index 00000000000000..979fbb7784c798 --- /dev/null +++ b/regression-test/suites/external_table_p0/paimon/test_paimon_partition_schema_filter_refs.groovy @@ -0,0 +1,366 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +import org.apache.doris.regression.action.ProfileAction + +suite("test_paimon_partition_schema_filter_refs", + "p0,external,paimon,external_docker,external_docker_paimon,nonConcurrent") { + String enabled = context.config.otherConfigs.get("enablePaimonTest") + if (enabled == null || !enabled.equalsIgnoreCase("true")) { + logger.info("disable paimon test") + return + } + + String minioPort = context.config.otherConfigs.get("iceberg_minio_port") + String externalEnvIp = context.config.otherConfigs.get("externalEnvIp") + String catalogName = "test_paimon_partition_schema_filter_refs" + String dbName = "paimon_partition_schema_filter_refs_db" + + def stringRows = { String query -> + sql(query).collect { row -> + row.collect { value -> value == null ? null : value.toString() } + } + } + def latestSnapshotId = { String tableName -> + return spark_paimon(""" + select snapshot_id + from paimon.${dbName}.`${tableName}\$snapshots` + order by snapshot_id desc + limit 1 + """)[0][0].toString() + } + def createTag = { String tableName, String tagName -> + spark_paimon """ + call paimon.sys.create_tag( + table => '${dbName}.${tableName}', + tag => '${tagName}' + ) + """ + } + def profileAction = new ProfileAction(context) + def profileCounterValues = { String profileText, String counterName -> + def values = [] + def matcher = profileText =~ ("(?m)^\\s*(?:-\\s*)?" + + java.util.regex.Pattern.quote(counterName) + ":\\s+([^\\n]+)") + while (matcher.find()) { + def number = matcher.group(1).toString() =~ /([0-9,]+)/ + if (number.find()) { + values.add(Long.parseLong(number.group(1).replace(",", ""))) + } + } + return values + } + def assertRuntimeFilterPruned = { String tableName, String dimensionTable, + List> expectedRows -> + String token = UUID.randomUUID().toString() + assertEquals(expectedRows, stringRows(""" + select /*+ SET_VAR(runtime_filter_type='IN_OR_BLOOM_FILTER') */ + '${token}', f.id + from ${tableName} f join ${dimensionTable} d on f.part = d.part + order by f.id + """).collect { row -> [row[1]] }) + // Scanner counters can arrive after the profile list first reports COMPLETE. + String profile = profileAction.getProfileBySql( + token, + ["RuntimeFilterPartitionPrunedRangeNum"], + 30000L, + 500L) + long fileRangesPruned = profileCounterValues( + profile, "RuntimeFilterPartitionPrunedRangeNum").sum(0L) + long partitionsPruned = profileCounterValues( + profile, "PartitionsPrunedByRuntimeFilter").sum(0L) + assertTrue(fileRangesPruned + partitionsPruned > 0L, + "Runtime filter did not prune any Paimon partition/file range; " + + profile.take(2000).replaceAll("\\s+", " ")) + } + def getExplainText = { String query -> + return sql("explain verbose ${query}").collect { row -> row[0].toString() }.join("\n") + } + def assertNativePath = { String query, String label -> + String explainText = getExplainText(query) + def splitMatcher = (explainText =~ /paimonNativeReadSplits=(\d+)\/(\d+)/) + assertTrue(splitMatcher.find(), "Expected paimonNativeReadSplits for ${label}") + long nativeSplits = Long.parseLong(splitMatcher.group(1)) + long totalSplits = Long.parseLong(splitMatcher.group(2)) + assertTrue(totalSplits > 0 && nativeSplits > 0, + "Expected native splits for ${label}, native=${nativeSplits}, total=${totalSplits}") + assertTrue(explainText.contains("SplitStat [type=NATIVE"), + "Expected a NATIVE split for ${label}") + } + def assertJniPath = { String query, String label -> + String explainText = getExplainText(query) + def splitMatcher = (explainText =~ /paimonNativeReadSplits=(\d+)\/(\d+)/) + assertTrue(splitMatcher.find(), "Expected paimonNativeReadSplits for ${label}") + long nativeSplits = Long.parseLong(splitMatcher.group(1)) + long totalSplits = Long.parseLong(splitMatcher.group(2)) + assertTrue(totalSplits > 0 && nativeSplits == 0, + "Expected JNI-only splits for ${label}, native=${nativeSplits}, total=${totalSplits}") + assertTrue(explainText.contains("SplitStat [type=JNI"), + "Expected a JNI split for ${label}") + } + + sql """drop catalog if exists ${catalogName}""" + sql """ + create catalog ${catalogName} properties ( + 'type'='paimon', + 'warehouse'='s3://warehouse/wh', + 's3.endpoint'='http://${externalEnvIp}:${minioPort}', + 's3.access_key'='admin', + 's3.secret_key'='password', + 's3.path.style.access'='true', + 'meta.cache.paimon.table.ttl-second'='0' + ) + """ + + try { + ["parquet", "orc"].each { String format -> + String tableName = "partition_schema_${format}" + String dimensionTable = "${tableName}_dimension" + spark_paimon_multi """ + create database if not exists paimon.${dbName}; + drop table if exists paimon.${dbName}.${tableName}; + create table paimon.${dbName}.${tableName} ( + id int, + part string, + payload struct, + attrs map>, + events array> + ) using paimon + partitioned by (part) + tblproperties ('file.format'='${format}'); + insert into paimon.${dbName}.${tableName} values + (1, 'p1', named_struct('metric', 10, 'label', 'base-1'), + map('k', named_struct('code', 100)), + array(named_struct('score', 1000))), + (2, 'p2', named_struct('metric', 20, 'label', 'base-2'), + map('k', named_struct('code', 200)), + array(named_struct('score', 2000))); + """ + String baseSnapshot = latestSnapshotId(tableName) + createTag(tableName, "${tableName}_base") + spark_paimon """ + call paimon.sys.create_branch( + '${dbName}.${tableName}', + '${tableName}_base_branch', + '${tableName}_base' + ) + """ + + // Scenario PM-PE01: Paimon keeps a fixed partition key while STRUCT, MAP-value + // STRUCT and ARRAY-element STRUCT children are added. + spark_paimon_multi """ + alter table paimon.${dbName}.${tableName} add column payload.extra string; + alter table paimon.${dbName}.${tableName} add column attrs.value.extra int; + alter table paimon.${dbName}.${tableName} add column events.element.extra int; + insert into paimon.${dbName}.${tableName} values + (3, 'p1', + named_struct('metric', 30, 'label', 'add-3', 'extra', 'payload-extra'), + map('k', named_struct('code', 300, 'extra', 301)), + array(named_struct('score', 3000, 'extra', 3001))), + (4, 'p3', + named_struct('metric', 40, 'label', 'add-4', 'extra', 'payload-extra'), + map('k', named_struct('code', 400, 'extra', 401)), + array(named_struct('score', 4000, 'extra', 4001))); + """ + String addedSnapshot = latestSnapshotId(tableName) + + // Scenario PM-PE02: rename and promote complex children without changing partition + // identity. Old snapshot/tag/branch schemas must remain independently readable. + spark_paimon_multi """ + alter table paimon.${dbName}.${tableName} + rename column payload.label to renamed_label; + alter table paimon.${dbName}.${tableName} + rename column attrs.value.code to renamed_code; + alter table paimon.${dbName}.${tableName} + rename column events.element.score to renamed_score; + alter table paimon.${dbName}.${tableName} + alter column payload.metric type bigint; + insert into paimon.${dbName}.${tableName} values + (5, 'p1', + named_struct('metric', 5000000000, + 'renamed_label', 'rename-5', 'extra', 'payload-extra'), + map('k', named_struct('renamed_code', 500, 'extra', 501)), + array(named_struct('renamed_score', 5000, 'extra', 5001))), + (6, 'p4', + named_struct('metric', 60, + 'renamed_label', 'rename-6', 'extra', 'payload-extra'), + map('k', named_struct('renamed_code', 600, 'extra', 601)), + array(named_struct('renamed_score', 6000, 'extra', 6001))); + """ + + // Scenario PM-PE03: drop/re-add a nested name to verify field IDs never leak values + // through partition-filtered scans. + spark_paimon_multi """ + alter table paimon.${dbName}.${tableName} drop column payload.extra; + alter table paimon.${dbName}.${tableName} add column payload.extra bigint; + insert into paimon.${dbName}.${tableName} values + (7, 'p1', + named_struct('metric', 70, + 'renamed_label', 'readd-7', 'extra', 7000), + map('k', named_struct('renamed_code', 700, 'extra', 701)), + array(named_struct('renamed_score', 7000, 'extra', 7001))); + drop table if exists paimon.${dbName}.${dimensionTable}; + create table paimon.${dbName}.${dimensionTable} (part string) + using paimon tblproperties ('file.format'='parquet'); + insert into paimon.${dbName}.${dimensionTable} values ('p1'); + """ + String finalSnapshot = latestSnapshotId(tableName) + createTag(tableName, "${tableName}_final") + + sql """switch ${catalogName}""" + sql """use ${dbName}""" + sql """refresh table ${tableName}""" + String actionSuffix = format + + // Scenario PM-F01: static partition filtering combines every complex-schema version. + "qt_${actionSuffix}_current_partition_filter"(""" + select id from ${tableName} where part = 'p1' order by id + """) + "qt_${actionSuffix}_current_promoted_metric"(""" + select id, payload.metric from ${tableName} + where part = 'p1' and payload.metric > 1000000000 + order by id + """) + "qt_${actionSuffix}_current_readded_payload"(""" + select id, payload.extra from ${tableName} + where part = 'p1' and payload.extra is not null + order by id + """) + // Project every evolved STRUCT/MAP/ARRAY child so a reader cannot pass by decoding + // only the top-level partition and payload fields. + "qt_${actionSuffix}_current_all_complex_children"(""" + select id, payload.renamed_label, payload.extra, + element_at(attrs, 'k').renamed_code, + element_at(attrs, 'k').extra, + events[1].renamed_score, + events[1].extra + from ${tableName} + where part = 'p1' + order by id + """) + + // Scenario PM-RF01: runtime-filter partition pruning stays result-equivalent across + // complex schema versions. + String rfQuery = """ + select /*+ SET_VAR(runtime_filter_type='IN_OR_BLOOM_FILTER') */ f.id + from ${tableName} f join ${dimensionTable} d on f.part = d.part + order by f.id + """ + sql """set runtime_filter_wait_infinitely=true""" + // Small fixture tables have no column statistics. Keep the generated RF so this + // suite validates scanner-side partition pruning rather than RF selectivity pruning. + sql """set enable_runtime_filter_prune=false""" + sql """set runtime_filter_mode=GLOBAL""" + sql """set parallel_pipeline_task_num=1""" + sql """set disable_join_reorder=true""" + sql """set enable_profile=true""" + sql """set profile_level=2""" + sql """set enable_runtime_filter_partition_prune=false""" + "qt_${actionSuffix}_rf_disabled"(rfQuery) + sql """set enable_runtime_filter_partition_prune=true""" + assertRuntimeFilterPruned( + tableName, dimensionTable, [["1"], ["3"], ["5"], ["7"]]) + + // Scenario PM-R01: numeric snapshot, tag and branch bind their historical schemas. + "qt_${actionSuffix}_base_snapshot"(""" + select id, part, payload.label + from ${tableName} for version as of ${baseSnapshot} + where part in ('p1', 'p2') order by id + """) + "qt_${actionSuffix}_base_tag"(""" + select id, part, payload.label + from ${tableName}@tag(${tableName}_base) + where part in ('p1', 'p2') order by id + """) + "qt_${actionSuffix}_base_branch"(""" + select id, part, payload.label + from ${tableName}@branch(${tableName}_base_branch) + where part in ('p1', 'p2') order by id + """) + "qt_${actionSuffix}_added_snapshot_complex"(""" + select id, payload.label, + element_at(attrs, 'k').code, + element_at(attrs, 'k').extra, + events[1].score, + events[1].extra + from ${tableName} for version as of ${addedSnapshot} + where part = 'p1' + order by id + """) + "qt_${actionSuffix}_final_tag"(""" + select id from ${tableName}@tag(${tableName}_final) + where part = 'p1' order by id + """) + "qt_${actionSuffix}_base_snapshot_complex"(""" + select id, element_at(attrs, 'k').code, events[1].score + from ${tableName} for version as of ${baseSnapshot} + order by id + """) + test { + sql """ + select element_at(attrs, 'k').renamed_code + from ${tableName} for version as of ${baseSnapshot} + """ + exception "renamed_code" + } + test { + sql """ + select events[1].score + from ${tableName}@tag(${tableName}_final) + """ + exception "score" + } + + // Scenario PM-RD01: JNI and native readers agree for partition-filtered historical + // and current complex projections, and the explain plan proves each requested path. + String currentReaderQuery = """ + select id, payload.renamed_label, payload.extra, + element_at(attrs, 'k').renamed_code, + events[1].renamed_score + from ${tableName} + where part in ('p1', 'p3') + order by id + """ + String historicalReaderQuery = """ + select id, payload.label, element_at(attrs, 'k').code, events[1].score + from ${tableName} for version as of ${baseSnapshot} + where part in ('p1', 'p2') + order by id + """ + sql """set enable_paimon_cpp_reader=false""" + sql """set force_jni_scanner=true""" + assertJniPath(currentReaderQuery, "${tableName} current") + assertJniPath(historicalReaderQuery, "${tableName} historical") + "qt_${actionSuffix}_jni_current_complex"(currentReaderQuery) + "qt_${actionSuffix}_jni_historical_complex"(historicalReaderQuery) + sql """set force_jni_scanner=false""" + sql """set enable_paimon_cpp_reader=true""" + assertNativePath(currentReaderQuery, "${tableName} current") + assertNativePath(historicalReaderQuery, "${tableName} historical") + "qt_${actionSuffix}_native_current_complex"(currentReaderQuery) + "qt_${actionSuffix}_native_historical_complex"(historicalReaderQuery) + assertEquals(finalSnapshot, latestSnapshotId(tableName)) + } + } finally { + sql """set enable_paimon_cpp_reader=false""" + sql """set force_jni_scanner=false""" + sql """set enable_runtime_filter_prune=true""" + sql """set enable_runtime_filter_partition_prune=true""" + sql """set disable_join_reorder=false""" + sql """drop catalog if exists ${catalogName}""" + } +} diff --git a/regression-test/suites/external_table_p0/paimon/test_paimon_schema_branch_partition_matrix.groovy b/regression-test/suites/external_table_p0/paimon/test_paimon_schema_branch_partition_matrix.groovy new file mode 100644 index 00000000000000..f0f795477396ef --- /dev/null +++ b/regression-test/suites/external_table_p0/paimon/test_paimon_schema_branch_partition_matrix.groovy @@ -0,0 +1,225 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +suite("test_paimon_schema_branch_partition_matrix", "p0,external,paimon") { + String enabled = context.config.otherConfigs.get("enablePaimonTest") + if (enabled == null || !enabled.equalsIgnoreCase("true")) { + logger.info("disable paimon test") + return + } + + String minioPort = context.config.otherConfigs.get("iceberg_minio_port") + String externalEnvIp = context.config.otherConfigs.get("externalEnvIp") + String catalogName = "test_paimon_schema_branch_partition_matrix" + String dbName = "paimon_schema_branch_partition_db" + String branchTable = "branch_schema_timeline" + String partitionTable = "partition_schema_timeline" + + sql """drop catalog if exists ${catalogName}""" + sql """ + create catalog ${catalogName} properties ( + 'type'='paimon', + 'warehouse'='s3://warehouse/wh', + 's3.endpoint'='http://${externalEnvIp}:${minioPort}', + 's3.access_key'='admin', + 's3.secret_key'='password', + 's3.path.style.access'='true', + 'meta.cache.paimon.table.ttl-second'='0' + ) + """ + + try { + spark_paimon_multi """ + create database if not exists paimon.${dbName}; + drop table if exists paimon.${dbName}.${branchTable}; + create table paimon.${dbName}.${branchTable} ( + id int, + old_name string, + metric int + ) using paimon + tblproperties ('file.format'='parquet'); + insert into paimon.${dbName}.${branchTable} values (1, 'base-1', 10); + call paimon.sys.create_tag( + table => '${dbName}.${branchTable}', + tag => 'branch_base' + ); + call paimon.sys.create_branch( + '${dbName}.${branchTable}', + 'schema_branch', + 'branch_base' + ); + """ + + // Scenario T09-branch-evolution: branch owns an independent rename/type/add timeline. + spark_paimon_multi """ + alter table paimon.${dbName}.`${branchTable}\$branch_schema_branch` + rename column old_name to branch_name; + alter table paimon.${dbName}.`${branchTable}\$branch_schema_branch` + alter column metric type bigint; + alter table paimon.${dbName}.`${branchTable}\$branch_schema_branch` + add column branch_only string; + insert into paimon.${dbName}.`${branchTable}\$branch_schema_branch` + values (2, 'branch-2', 6000000000, 'branch-only-2'); + """ + + // Main evolves independently after the branch is created. + spark_paimon_multi """ + alter table paimon.${dbName}.${branchTable} add column main_only string; + insert into paimon.${dbName}.${branchTable} + values (3, 'main-3', 30, 'main-only-3'); + """ + + sql """switch ${catalogName}""" + sql """use ${dbName}""" + sql """refresh table ${branchTable}""" + + assertEquals([[1, "base-1", 10, null], [3, "main-3", 30, "main-only-3"]], sql(""" + select id, old_name, metric, main_only + from ${branchTable} + order by id + """)) + // Negative contract: Doris cannot initialize a Paimon branch with an independent schema. + test { + sql """ + select id, branch_name, metric, branch_only + from ${branchTable}@branch(schema_branch) + order by id + """ + exception "failed to initSchema" + } + test { + sql """select branch_only from ${branchTable}""" + exception "Unknown column 'branch_only'" + } + + // Scenario T09-fast-forward: branch schema and data replace main after Paimon fast-forward. + spark_paimon """ + call paimon.sys.fast_forward('${dbName}.${branchTable}', 'schema_branch') + """ + spark_paimon """refresh table paimon.${dbName}.${branchTable}""" + sql """refresh table ${branchTable}""" + assertEquals([ + [1, "base-1", 10L, null], + [2, "branch-2", 6000000000L, "branch-only-2"] + ], spark_paimon(""" + select id, branch_name, metric, branch_only + from paimon.${dbName}.${branchTable} + order by id + """)) + List fastForwardColumns = sql("""desc ${branchTable}""") + .collect { row -> row[0].toString() } + assertTrue(fastForwardColumns.containsAll( + ["id", "branch_name", "metric", "branch_only"])) + test { + sql """select old_name from ${branchTable}""" + exception "Unknown column 'old_name'" + } + + spark_paimon_multi """ + drop table if exists paimon.${dbName}.${partitionTable}; + create table paimon.${dbName}.${partitionTable} ( + id int, + part string, + old_payload string + ) using paimon + partitioned by (part) + tblproperties ('file.format'='orc'); + insert into paimon.${dbName}.${partitionTable} + values (1, 'p1', 'old-1'), (2, 'p2', 'old-2'); + call paimon.sys.create_tag( + table => '${dbName}.${partitionTable}', + tag => 'partition_before_change' + ); + alter table paimon.${dbName}.${partitionTable} + rename column old_payload to new_payload; + insert into paimon.${dbName}.${partitionTable} + values (3, 'p3', 'new-3'); + """ + sql """refresh table ${partitionTable}""" + + // Scenario S16-supported: non-partition payload rename keeps old/new snapshot bindings. + assertEquals([[1, "p1", "old-1"], [2, "p2", "old-2"]], sql(""" + select id, part, old_payload + from ${partitionTable}@tag(partition_before_change) + order by id + """)) + assertEquals([ + [1, "p1", "old-1"], + [2, "p2", "old-2"], + [3, "p3", "new-3"] + ], sql(""" + select id, part, new_payload from ${partitionTable} order by id + """)) + + // Scenario S16-negative: Paimon partition keys only support reordering. + // The operation must fail atomically without changing schema, snapshots or partition data. + int snapshotsBefore = spark_paimon(""" + select count(*) + from paimon.${dbName}.`${partitionTable}\$snapshots` + """)[0][0].toString().toInteger() + boolean renameRejected = false + try { + spark_paimon """ + alter table paimon.${dbName}.${partitionTable} + rename column part to renamed_part + """ + } catch (Exception e) { + renameRejected = true + assertTrue(e.message.toLowerCase().contains("partition")) + } + assertTrue(renameRejected, "Paimon must reject partition-key rename") + + boolean typeRejected = false + try { + spark_paimon """ + alter table paimon.${dbName}.${partitionTable} + alter column part type bigint + """ + } catch (Exception e) { + typeRejected = true + assertTrue(e.message.toLowerCase().contains("partition")) + } + assertTrue(typeRejected, "Paimon must reject partition-key type changes") + + boolean dropRejected = false + try { + spark_paimon """ + alter table paimon.${dbName}.${partitionTable} drop column part + """ + } catch (Exception e) { + dropRejected = true + assertTrue(e.message.toLowerCase().contains("partition")) + } + assertTrue(dropRejected, "Paimon must reject dropping a partition key") + + int snapshotsAfter = spark_paimon(""" + select count(*) + from paimon.${dbName}.`${partitionTable}\$snapshots` + """)[0][0].toString().toInteger() + assertEquals(snapshotsBefore, snapshotsAfter) + sql """refresh table ${partitionTable}""" + assertEquals([ + [1, "p1", "old-1"], + [2, "p2", "old-2"], + [3, "p3", "new-3"] + ], sql(""" + select id, part, new_payload from ${partitionTable} order by id + """)) + } finally { + sql """drop catalog if exists ${catalogName}""" + } +} diff --git a/regression-test/suites/external_table_p0/paimon/test_paimon_schema_dual_relation_matrix.groovy b/regression-test/suites/external_table_p0/paimon/test_paimon_schema_dual_relation_matrix.groovy new file mode 100644 index 00000000000000..b5478dfbea7348 --- /dev/null +++ b/regression-test/suites/external_table_p0/paimon/test_paimon_schema_dual_relation_matrix.groovy @@ -0,0 +1,188 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +suite("test_paimon_schema_dual_relation_matrix", "p0,external,paimon") { + String enabled = context.config.otherConfigs.get("enablePaimonTest") + if (enabled == null || !enabled.equalsIgnoreCase("true")) { + logger.info("disable paimon test") + return + } + + String minioPort = context.config.otherConfigs.get("iceberg_minio_port") + String externalEnvIp = context.config.otherConfigs.get("externalEnvIp") + String catalogName = "test_paimon_schema_dual_relation_matrix" + String dbName = "paimon_schema_dual_relation_db" + String tableName = "dual_schema_timeline" + + def latestSnapshotId = { + List> rows = spark_paimon """ + select snapshot_id + from paimon.${dbName}.`${tableName}\$snapshots` + order by snapshot_id desc + limit 1 + """ + assertEquals(1, rows.size()) + return rows[0][0].toString() + } + + sql """drop catalog if exists ${catalogName}""" + sql """ + create catalog ${catalogName} properties ( + 'type'='paimon', + 'warehouse'='s3://warehouse/wh', + 's3.endpoint'='http://${externalEnvIp}:${minioPort}', + 's3.access_key'='admin', + 's3.secret_key'='password', + 's3.path.style.access'='true' + ) + """ + + try { + spark_paimon_multi """ + create database if not exists paimon.${dbName}; + drop table if exists paimon.${dbName}.${tableName}; + create table paimon.${dbName}.${tableName} ( + id int, + old_name string, + info struct + ) using paimon + tblproperties ('file.format'='parquet'); + insert into paimon.${dbName}.${tableName} + values (1, 'old-1', named_struct('added', 10, 'keep', 11)); + """ + String oldSnapshot = latestSnapshotId() + + spark_paimon_multi """ + alter table paimon.${dbName}.${tableName} rename column old_name to new_name; + alter table paimon.${dbName}.${tableName} + rename column info.added to renamed; + insert into paimon.${dbName}.${tableName} + values (2, 'new-2', named_struct('renamed', 20, 'keep', 21)); + """ + String newSnapshot = latestSnapshotId() + + sql """switch ${catalogName}""" + sql """use ${dbName}""" + + // Scenario TC07-baseline: verify a single historical relation binds its own schema. + assertEquals([[1, "old-1", 10]], sql(""" + select id, old_name, info.added + from ${tableName} for version as of ${oldSnapshot} + order by id + """)) + assertEquals([[1, "old-1", 10], [2, "new-2", 20]], sql(""" + select id, new_name, info.renamed + from ${tableName} for version as of ${newSnapshot} + order by id + """)) + + // Scenario TC07-join negative contract: + // two Paimon historical relations currently reuse the first schema. + test { + sql """ + select o.id, o.old_name, n.new_name + from ( + select id, old_name + from ${tableName} for version as of ${oldSnapshot} + ) o + join ( + select id, new_name + from ${tableName} for version as of ${newSnapshot} + ) n on o.id = n.id + order by o.id + """ + exception "Unknown column 'new_name'" + } + + // Scenario TC07-reverse-join: binding must be independent of relation order. + test { + sql """ + select n.id, n.new_name, o.old_name + from ( + select id, new_name + from ${tableName} for version as of ${newSnapshot} + ) n + join ( + select id, old_name + from ${tableName} for version as of ${oldSnapshot} + ) o on n.id = o.id + order by n.id + """ + exception "Unknown column 'old_name'" + } + + // Scenario TC07-union: top-level historical schemas stay relation-local. + test { + sql """ + select id, old_name as name_value + from ${tableName} for version as of ${oldSnapshot} + union all + select id, new_name as name_value + from ${tableName} for version as of ${newSnapshot} + order by id, name_value + """ + exception "Unknown column 'new_name'" + } + + // Scenario TC07-nested-union: nested lookup is also relation-local. + test { + sql """ + select id, info.added as nested_value + from ${tableName} for version as of ${oldSnapshot} + union all + select id, info.renamed as nested_value + from ${tableName} for version as of ${newSnapshot} + order by id, nested_value + """ + exception "No such struct field 'renamed'" + } + + // Scenario TC07-CTE: CTE boundaries must not collapse snapshot schemas. + test { + sql """ + with old_ref as ( + select id, old_name + from ${tableName} for version as of ${oldSnapshot} + ), new_ref as ( + select id, new_name + from ${tableName} for version as of ${newSnapshot} + ) + select old_ref.id, old_ref.old_name, new_ref.new_name + from old_ref join new_ref on old_ref.id = new_ref.id + order by old_ref.id + """ + exception "Unknown column 'new_name'" + } + + // Scenario TC07-correlated-subquery: subqueries require an independent schema. + test { + sql """ + select o.id, o.old_name + from ${tableName} for version as of ${oldSnapshot} o + where exists ( + select 1 + from ${tableName} for version as of ${newSnapshot} n + where n.id = o.id and n.new_name is not null + ) + order by o.id + """ + exception "Unknown column 'new_name'" + } + } finally { + sql """drop catalog if exists ${catalogName}""" + } +} diff --git a/regression-test/suites/external_table_p0/paimon/test_paimon_schema_metadata_atomicity_matrix.groovy b/regression-test/suites/external_table_p0/paimon/test_paimon_schema_metadata_atomicity_matrix.groovy new file mode 100644 index 00000000000000..5672607cf1750a --- /dev/null +++ b/regression-test/suites/external_table_p0/paimon/test_paimon_schema_metadata_atomicity_matrix.groovy @@ -0,0 +1,170 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +suite("test_paimon_schema_metadata_atomicity_matrix", "p0,external,paimon") { + String enabled = context.config.otherConfigs.get("enablePaimonTest") + if (enabled == null || !enabled.equalsIgnoreCase("true")) { + logger.info("disable paimon test") + return + } + + String minioPort = context.config.otherConfigs.get("iceberg_minio_port") + String externalEnvIp = context.config.otherConfigs.get("externalEnvIp") + String catalogName = "test_paimon_schema_metadata_atomicity_matrix" + String dbName = "paimon_schema_metadata_atomicity_db" + String tableName = "metadata_timeline" + + def createTag = { String tagName -> + spark_paimon """ + call paimon.sys.create_tag( + table => '${dbName}.${tableName}', + tag => '${tagName}' + ) + """ + } + def snapshotCount = { + return spark_paimon(""" + select count(*) from paimon.${dbName}.`${tableName}\$snapshots` + """)[0][0].toString().toInteger() + } + def schemaCount = { + return spark_paimon(""" + select count(*) from paimon.${dbName}.`${tableName}\$schemas` + """)[0][0].toString().toInteger() + } + + sql """drop catalog if exists ${catalogName}""" + sql """ + create catalog ${catalogName} properties ( + 'type'='paimon', + 'warehouse'='s3://warehouse/wh', + 's3.endpoint'='http://${externalEnvIp}:${minioPort}', + 's3.access_key'='admin', + 's3.secret_key'='password', + 's3.path.style.access'='true', + 'meta.cache.paimon.table.ttl-second'='0' + ) + """ + + try { + spark_paimon_multi """ + create database if not exists paimon.${dbName}; + drop table if exists paimon.${dbName}.${tableName}; + create table paimon.${dbName}.${tableName} ( + id int not null, + required_name string not null, + optional_value int, + info struct + ) using paimon + tblproperties ('file.format'='parquet'); + insert into paimon.${dbName}.${tableName} + values (1, 'name-1', 10, named_struct('value', 100, 'keep', 101)); + """ + createTag("metadata_before_change") + int initialSnapshots = snapshotCount() + + // Scenario S19-comment: top-level and nested comments create schema versions, not data snapshots. + spark_paimon """ + alter table paimon.${dbName}.${tableName} + alter column required_name comment 'top-level-comment' + """ + spark_paimon """ + alter table paimon.${dbName}.${tableName} + alter column info.value comment 'nested-comment' + """ + assertEquals(initialSnapshots, snapshotCount()) + + // Scenario S19-nullability: required-to-optional keeps both current and tagged reads stable. + spark_paimon """ + alter table paimon.${dbName}.${tableName} + alter column required_name drop not null + """ + sql """switch ${catalogName}""" + sql """use ${dbName}""" + sql """refresh table ${tableName}""" + assertEquals([[1, "name-1", 10, 100]], sql(""" + select id, required_name, optional_value, info.value + from ${tableName} + order by id + """)) + assertEquals([[1, "name-1", 10, 100]], sql(""" + select id, required_name, optional_value, info.value + from ${tableName}@tag(metadata_before_change) + order by id + """)) + + // Scenario S20-default: defaults are schema changes and apply when a later insert omits the field. + int schemasBeforeDefault = schemaCount() + spark_paimon """ + alter table paimon.${dbName}.${tableName} + alter column optional_value set default 7 + """ + assertTrue(schemaCount() > schemasBeforeDefault) + spark_paimon """ + insert into paimon.${dbName}.${tableName} (id, required_name, info) + values (2, 'name-2', named_struct('value', 200, 'keep', 201)) + """ + int snapshotsAfterDefault = snapshotCount() + sql """refresh table ${tableName}""" + assertEquals([[1, 10], [2, 7]], sql(""" + select id, optional_value from ${tableName} order by id + """)) + + // Scenario S20-nullability-strengthen: Paimon rejects optional-to-required atomically. + int schemasBeforeNotNull = schemaCount() + boolean notNullRejected = false + try { + spark_paimon """ + alter table paimon.${dbName}.${tableName} + alter column optional_value set not null + """ + } catch (Exception e) { + notNullRejected = true + assertTrue(e.message.contains("Cannot change nullable column to non-nullable")) + } + assertTrue(notNullRejected, "Paimon must reject optional-to-required evolution") + assertEquals(schemasBeforeNotNull, schemaCount()) + + // Scenario S20-narrowing: incompatible narrowing is atomic. + int schemasBeforeNarrowing = schemaCount() + boolean narrowingRejected = false + try { + spark_paimon """ + alter table paimon.${dbName}.${tableName} + alter column optional_value type smallint + """ + } catch (Exception e) { + narrowingRejected = true + assertTrue(e.message.toLowerCase().contains("type")) + } + assertTrue(narrowingRejected, "Paimon must reject incompatible narrowing") + assertEquals(schemasBeforeNarrowing, schemaCount()) + + sql """refresh table ${tableName}""" + assertEquals([ + [1, "name-1", 10, 100], + [2, "name-2", 7, 200] + ], sql(""" + select id, required_name, optional_value, info.value + from ${tableName} + order by id + """)) + assertEquals(snapshotsAfterDefault, snapshotCount()) + } finally { + sql """drop catalog if exists ${catalogName}""" + } +} diff --git a/regression-test/suites/external_table_p0/paimon/test_paimon_schema_time_travel_matrix.groovy b/regression-test/suites/external_table_p0/paimon/test_paimon_schema_time_travel_matrix.groovy new file mode 100644 index 00000000000000..7ec5a571a211df --- /dev/null +++ b/regression-test/suites/external_table_p0/paimon/test_paimon_schema_time_travel_matrix.groovy @@ -0,0 +1,600 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +suite("test_paimon_schema_time_travel_matrix", "p0,external,paimon") { + String enabled = context.config.otherConfigs.get("enablePaimonTest") + if (enabled == null || !enabled.equalsIgnoreCase("true")) { + logger.info("disable paimon test") + return + } + + String minioPort = context.config.otherConfigs.get("iceberg_minio_port") + String externalEnvIp = context.config.otherConfigs.get("externalEnvIp") + String catalogName = "test_paimon_schema_time_travel_matrix" + String noCacheCatalogName = "test_paimon_schema_time_travel_matrix_no_cache" + String dbName = "paimon_schema_time_travel_matrix_db" + String topTable = "top_timeline" + String nestedTable = "nested_timeline" + String pkTable = "pk_dv_timeline" + String partitionTable = "partition_timeline" + + def latestSnapshotId = { String tableName -> + List> rows = spark_paimon """ + SELECT snapshot_id + FROM paimon.${dbName}.`${tableName}\$snapshots` + ORDER BY snapshot_id DESC + LIMIT 1 + """ + assertEquals(1, rows.size()) + return rows[0][0].toString() + } + + def createTag = { String tableName, String tagName, String snapshotId -> + spark_paimon """ + CALL paimon.sys.create_tag( + table => '${dbName}.${tableName}', + tag => '${tagName}', + snapshot => ${snapshotId} + ) + """ + } + + def assertUnknownColumn = { String query, String columnName -> + test { + sql query + exception "'${columnName}'" + } + } + + sql """drop catalog if exists ${catalogName}""" + sql """drop catalog if exists ${noCacheCatalogName}""" + sql """ + CREATE CATALOG ${catalogName} PROPERTIES ( + 'type'='paimon', + 'warehouse'='s3://warehouse/wh', + 's3.endpoint'='http://${externalEnvIp}:${minioPort}', + 's3.access_key'='admin', + 's3.secret_key'='password', + 's3.path.style.access'='true' + ) + """ + sql """ + CREATE CATALOG ${noCacheCatalogName} PROPERTIES ( + 'type'='paimon', + 'warehouse'='s3://warehouse/wh', + 's3.endpoint'='http://${externalEnvIp}:${minioPort}', + 's3.access_key'='admin', + 's3.secret_key'='password', + 's3.path.style.access'='true', + 'meta.cache.paimon.table.ttl-second'='0' + ) + """ + + try { + spark_paimon_multi """ + CREATE DATABASE IF NOT EXISTS paimon.${dbName}; + DROP TABLE IF EXISTS paimon.${dbName}.${topTable}; + CREATE TABLE paimon.${dbName}.${topTable} ( + id INT, + old_name STRING, + victim STRING, + metric INT + ) USING paimon + TBLPROPERTIES ('file.format'='parquet'); + INSERT INTO paimon.${dbName}.${topTable} + VALUES (1, 'alpha', 'old-v1', 10); + """ + String topCp0 = latestSnapshotId(topTable) + createTag(topTable, "top_cp0", topCp0) + spark_paimon """ + CALL paimon.sys.create_branch( + '${dbName}.${topTable}', + 'top_cp0_branch', + 'top_cp0' + ) + """ + Thread.sleep(1100) + + // Scenario S01/S02/S03 x T00/T01/T02/T05/T08: + // add multiple columns and position one AFTER old_name. + spark_paimon_multi """ + ALTER TABLE paimon.${dbName}.${topTable} + ADD COLUMNS (added STRING AFTER old_name, added_second BIGINT); + INSERT INTO paimon.${dbName}.${topTable} + (id, old_name, victim, metric, added, added_second) + VALUES (2, 'beta', 'old-v2', 20, 'added-v2', 200); + """ + String topCpAdd = latestSnapshotId(topTable) + createTag(topTable, "top_cp_add", topCpAdd) + Thread.sleep(1100) + + // Scenario S04/S05 x T00-T08: + // explicit old/new column binding catches rename regressions hidden by SELECT *. + spark_paimon_multi """ + ALTER TABLE paimon.${dbName}.${topTable} + RENAME COLUMN old_name TO MixedName; + INSERT INTO paimon.${dbName}.${topTable} + (id, MixedName, victim, metric, added, added_second) + VALUES (3, 'gamma', 'old-v3', 30, 'added-v3', 300); + """ + String topCpRename = latestSnapshotId(topTable) + createTag(topTable, "top_cp_rename", topCpRename) + Thread.sleep(1100) + + // Scenario S06 x T00/T01/T02/T05: the dropped field remains available only to old refs. + spark_paimon_multi """ + ALTER TABLE paimon.${dbName}.${topTable} DROP COLUMN victim; + INSERT INTO paimon.${dbName}.${topTable} + (id, MixedName, metric, added, added_second) + VALUES (4, 'delta', 40, 'added-v4', 400); + """ + String topCpDrop = latestSnapshotId(topTable) + createTag(topTable, "top_cp_drop", topCpDrop) + Thread.sleep(1100) + + // Scenario S07 x T00/T01/T02/T05/T12/T13: + // drop/re-add with a different type creates a new field ID and must not expose old values. + spark_paimon_multi """ + ALTER TABLE paimon.${dbName}.${topTable} ADD COLUMN victim BIGINT; + INSERT INTO paimon.${dbName}.${topTable} + (id, MixedName, metric, added, added_second, victim) + VALUES (5, 'epsilon', 50, 'added-v5', 500, 5000); + """ + String topCpReadd = latestSnapshotId(topTable) + createTag(topTable, "top_cp_readd", topCpReadd) + Thread.sleep(1100) + + // Scenario S08 x T00/T01/T02/T05: compatible INT -> BIGINT promotion. + spark_paimon_multi """ + ALTER TABLE paimon.${dbName}.${topTable} ALTER COLUMN metric TYPE BIGINT; + INSERT INTO paimon.${dbName}.${topTable} + (id, MixedName, metric, added, added_second, victim) + VALUES (6, 'zeta', 6000000000, 'added-v6', 600, 6000); + """ + String topCpPromote = latestSnapshotId(topTable) + createTag(topTable, "top_cp_promote", topCpPromote) + + spark_paimon_multi """ + DROP TABLE IF EXISTS paimon.${dbName}.${nestedTable}; + CREATE TABLE paimon.${dbName}.${nestedTable} ( + id INT, + payload STRUCT, + attributes MAP>, + events ARRAY> + ) USING paimon + TBLPROPERTIES ('file.format'='orc'); + INSERT INTO paimon.${dbName}.${nestedTable} VALUES ( + 1, + named_struct('old_child', 10, 'keep', 11), + map('a', named_struct('old_child', 20, 'keep', 21)), + array(named_struct('old_child', 30, 'keep', 31)) + ); + """ + String nestedCp0 = latestSnapshotId(nestedTable) + createTag(nestedTable, "nested_cp0", nestedCp0) + + // Scenario S09/S14/S15 x T00/T01/T02/T05: + // add children to STRUCT, MAP value struct and ARRAY element struct. + spark_paimon_multi """ + ALTER TABLE paimon.${dbName}.${nestedTable} ADD COLUMN payload.added_child INT; + ALTER TABLE paimon.${dbName}.${nestedTable} + ADD COLUMN attributes.value.added_child INT; + ALTER TABLE paimon.${dbName}.${nestedTable} + ADD COLUMN events.element.added_child INT; + INSERT INTO paimon.${dbName}.${nestedTable} VALUES ( + 2, + named_struct('old_child', 110, 'keep', 111, 'added_child', 112), + map('a', named_struct('old_child', 120, 'keep', 121, 'added_child', 122)), + array(named_struct('old_child', 130, 'keep', 131, 'added_child', 132)) + ); + """ + String nestedCpAdd = latestSnapshotId(nestedTable) + + // Scenario S10/S14/S15 x T00/T01/T02/T05: rename every supported nested path. + spark_paimon_multi """ + ALTER TABLE paimon.${dbName}.${nestedTable} + RENAME COLUMN payload.old_child TO renamed_child; + ALTER TABLE paimon.${dbName}.${nestedTable} + RENAME COLUMN attributes.value.old_child TO renamed_child; + ALTER TABLE paimon.${dbName}.${nestedTable} + RENAME COLUMN events.element.old_child TO renamed_child; + INSERT INTO paimon.${dbName}.${nestedTable} VALUES ( + 3, + named_struct('renamed_child', 210, 'keep', 211, 'added_child', 212), + map('a', named_struct('renamed_child', 220, 'keep', 221, 'added_child', 222)), + array(named_struct('renamed_child', 230, 'keep', 231, 'added_child', 232)) + ); + """ + String nestedCpRename = latestSnapshotId(nestedTable) + createTag(nestedTable, "nested_cp_rename", nestedCpRename) + + // Scenario S11/S12/S13 x T00/T01/T02/T05: + // nested drop/re-add and type promotion preserve field-ID isolation. + spark_paimon_multi """ + ALTER TABLE paimon.${dbName}.${nestedTable} DROP COLUMN payload.renamed_child; + ALTER TABLE paimon.${dbName}.${nestedTable} + DROP COLUMN attributes.value.renamed_child; + ALTER TABLE paimon.${dbName}.${nestedTable} + DROP COLUMN events.element.renamed_child; + ALTER TABLE paimon.${dbName}.${nestedTable} ADD COLUMN payload.renamed_child BIGINT; + ALTER TABLE paimon.${dbName}.${nestedTable} + ADD COLUMN attributes.value.renamed_child BIGINT; + ALTER TABLE paimon.${dbName}.${nestedTable} + ADD COLUMN events.element.renamed_child BIGINT; + ALTER TABLE paimon.${dbName}.${nestedTable} + ALTER COLUMN payload.keep TYPE BIGINT; + INSERT INTO paimon.${dbName}.${nestedTable} VALUES ( + 4, + named_struct('keep', 311, 'added_child', 312, 'renamed_child', 3100), + map('a', named_struct('keep', 321, 'added_child', 322, 'renamed_child', 3200)), + array(named_struct('keep', 331, 'added_child', 332, 'renamed_child', 3300)) + ); + """ + String nestedCpReadd = latestSnapshotId(nestedTable) + + spark_paimon_multi """ + DROP TABLE IF EXISTS paimon.${dbName}.${pkTable}; + CREATE TABLE paimon.${dbName}.${pkTable} ( + id INT NOT NULL, + old_name STRING, + note STRING, + score INT + ) USING paimon + TBLPROPERTIES ( + 'bucket'='1', + 'primary-key'='id', + 'file.format'='parquet', + 'deletion-vectors.enabled'='true' + ); + INSERT INTO paimon.${dbName}.${pkTable} VALUES + (1, 'alpha', 'old-note-1', 10), + (2, 'beta', 'old-note-2', 20); + """ + String pkCp0 = latestSnapshotId(pkTable) + createTag(pkTable, "pk_cp0", pkCp0) + + // Scenario S01/S04/S18 x TC05: + // PK remains stable while a non-key field is added/renamed and rows are upserted/deleted. + spark_paimon_multi """ + ALTER TABLE paimon.${dbName}.${pkTable} ADD COLUMN extra STRING; + ALTER TABLE paimon.${dbName}.${pkTable} RENAME COLUMN old_name TO full_name; + INSERT INTO paimon.${dbName}.${pkTable} + (id, full_name, note, score, extra) + VALUES (1, 'alpha-updated', 'new-note-1', 11, 'extra-1'); + DELETE FROM paimon.${dbName}.${pkTable} WHERE id = 2; + """ + String pkCpRenameDelete = latestSnapshotId(pkTable) + createTag(pkTable, "pk_cp_rename_delete", pkCpRenameDelete) + + // Scenario S06/S07/S08/S18 x TC05: + // drop/re-add and promotion are combined with a later upsert and DV compaction. + spark_paimon_multi """ + ALTER TABLE paimon.${dbName}.${pkTable} DROP COLUMN note; + ALTER TABLE paimon.${dbName}.${pkTable} ADD COLUMN note BIGINT; + ALTER TABLE paimon.${dbName}.${pkTable} ALTER COLUMN score TYPE BIGINT; + INSERT INTO paimon.${dbName}.${pkTable} + (id, full_name, score, extra, note) + VALUES (3, 'gamma', 3000000000, 'extra-3', 3000); + """ + String pkCpReadd = latestSnapshotId(pkTable) + createTag(pkTable, "pk_cp_readd", pkCpReadd) + spark_paimon """ + CALL paimon.sys.compact(table => '${dbName}.${pkTable}', compact_strategy => 'full') + """ + + spark_paimon_multi """ + DROP TABLE IF EXISTS paimon.${dbName}.${partitionTable}; + CREATE TABLE paimon.${dbName}.${partitionTable} ( + id INT, + old_partition STRING, + payload STRING + ) USING paimon + PARTITIONED BY (old_partition) + TBLPROPERTIES ('file.format'='parquet'); + INSERT INTO paimon.${dbName}.${partitionTable} + VALUES (1, 'p1', 'old'), (2, 'p2', 'old'); + """ + String partitionCp0 = latestSnapshotId(partitionTable) + createTag(partitionTable, "partition_cp0", partitionCp0) + + // Scenario S16-negative: Paimon must reject partition-column rename atomically. + String partitionRenameError = null + try { + spark_paimon """ + ALTER TABLE paimon.${dbName}.${partitionTable} + RENAME COLUMN old_partition TO new_partition + """ + } catch (Exception e) { + partitionRenameError = e.getMessage() + } + assertNotNull(partitionRenameError) + assertTrue(partitionRenameError.contains("Cannot rename partition column")) + + // Scenario S17 x TC03: rename a payload field while retaining partition pruning. + spark_paimon_multi """ + ALTER TABLE paimon.${dbName}.${partitionTable} + RENAME COLUMN payload TO new_payload; + INSERT INTO paimon.${dbName}.${partitionTable} + VALUES (3, 'p3', 'new'); + """ + String partitionCpRename = latestSnapshotId(partitionTable) + + sql """switch ${catalogName}""" + sql """use ${dbName}""" + sql """refresh catalog ${catalogName}""" + + // Scenario TC01: validate latest schema/data, explicit new binding, predicate and aggregate. + assertEquals([[1, null], [2, null], [3, null], [4, null], [5, 5000L], [6, 6000L]], + sql("""select id, victim from ${topTable} order by id""")) + assertEquals([[6, 6000000000L]], + sql("""select id, metric from ${topTable} where metric > 5000000000""")) + assertEquals([[6L, 6000000150L]], + sql("""select count(*), sum(metric) from ${topTable}""")) + assertUnknownColumn("""select old_name from ${topTable}""", "old_name") + + // Scenario T01/T05/T06/T08: old snapshot/tag/branch uses the old schema. + List> topCp0Rows = [[1, "alpha", "old-v1", 10]] + assertEquals(topCp0Rows, sql(""" + select id, old_name, victim, metric + from ${topTable} for version as of ${topCp0} + order by id + """)) + assertEquals(topCp0Rows, sql(""" + select id, old_name, victim, metric + from ${topTable} for version as of 'top_cp0' + order by id + """)) + assertEquals(topCp0Rows, sql(""" + select id, old_name, victim, metric + from ${topTable}@tag(top_cp0) + order by id + """)) + assertEquals(topCp0Rows, sql(""" + select id, old_name, victim, metric + from ${topTable}@branch(top_cp0_branch) + order by id + """)) + assertUnknownColumn(""" + select MixedName from ${topTable} for version as of ${topCp0} + """, "MixedName") + + // Scenario T03/T04: time string and epoch millis resolve to the pre-change schema. + List> cp0TimeRows = sql(""" + select date_format(date_add(commit_time, interval 1 second), '%Y-%m-%d %H:%i:%s'), + cast(unix_timestamp(commit_time) * 1000 + 999 as bigint) + from ${topTable}\$snapshots + where snapshot_id = ${topCp0} + """) + assertEquals(1, cp0TimeRows.size()) + assertEquals(topCp0Rows, sql(""" + select id, old_name, victim, metric + from ${topTable} for time as of "${cp0TimeRows[0][0]}" + order by id + """)) + assertEquals(topCp0Rows, sql(""" + select id, old_name, victim, metric + from ${topTable} for time as of ${cp0TimeRows[0][1]} + order by id + """)) + + // Scenario S01-S08: checkpoint projections prove name/type/field-ID isolation. + assertEquals([[1, "alpha", null], [2, "beta", "added-v2"]], + sql(""" + select id, old_name, added + from ${topTable} for version as of ${topCpAdd} + order by id + """)) + assertEquals([[1, "alpha"], [2, "beta"], [3, "gamma"]], + sql(""" + select id, MixedName + from ${topTable} for version as of ${topCpRename} + where metric >= 10 + order by id + """)) + assertUnknownColumn(""" + select old_name from ${topTable} for version as of ${topCpRename} + """, "old_name") + assertUnknownColumn(""" + select victim from ${topTable} for version as of ${topCpDrop} + """, "victim") + assertEquals([[1, null], [2, null], [3, null], [4, null], [5, 5000L]], + sql(""" + select id, victim + from ${topTable} for version as of ${topCpReadd} + order by id + """)) + assertEquals([[6, 6000000000L]], + sql(""" + select id, metric + from ${topTable} for version as of ${topCpPromote} + where metric > 5000000000 + """)) + + // Scenario TC02: nested refs validate STRUCT/MAP/ARRAY projection and predicate. + assertEquals([[1, 10, 20, 30]], + sql(""" + select id, payload.old_child, + element_at(attributes, 'a').old_child, + events[1].old_child + from ${nestedTable} for version as of ${nestedCp0} + where payload.old_child = 10 + """)) + assertEquals([[1, null, null, null], [2, 112, 122, 132]], + sql(""" + select id, payload.added_child, + element_at(attributes, 'a').added_child, + events[1].added_child + from ${nestedTable} for version as of ${nestedCpAdd} + order by id + """)) + assertEquals([[1, 10, 20, 30], [2, 110, 120, 130], [3, 210, 220, 230]], + sql(""" + select id, payload.renamed_child, + element_at(attributes, 'a').renamed_child, + events[1].renamed_child + from ${nestedTable} for version as of 'nested_cp_rename' + order by id + """)) + assertUnknownColumn(""" + select payload.old_child + from ${nestedTable} for version as of ${nestedCpRename} + """, "old_child") + assertEquals([[1, null], [2, null], [3, null], [4, 3100L]], + sql(""" + select id, payload.renamed_child + from ${nestedTable} for version as of ${nestedCpReadd} + order by id + """)) + + // Scenario TC05/S18: PK upsert/delete/DV results remain correct at old and new refs. + assertEquals([[1, "alpha", "old-note-1", 10], [2, "beta", "old-note-2", 20]], + sql(""" + select id, old_name, note, score + from ${pkTable} for version as of 'pk_cp0' + order by id + """)) + assertEquals([[1, "alpha-updated", "new-note-1", 11, "extra-1"]], + sql(""" + select id, full_name, note, score, extra + from ${pkTable} for version as of ${pkCpRenameDelete} + order by id + """)) + assertEquals([[1, "alpha-updated", null], [3, "gamma", 3000L]], + sql(""" + select id, full_name, note + from ${pkTable} for version as of ${pkCpReadd} + order by id + """)) + assertUnknownColumn(""" + select old_name from ${pkTable} for version as of ${pkCpRenameDelete} + """, "old_name") + + // Scenario T14: incremental reads crossing a rename checkpoint bind the end schema. + List> incrementalJni + List> incrementalCpp + sql """set force_jni_scanner=false""" + sql """set enable_paimon_cpp_reader=false""" + incrementalJni = sql(""" + select id, full_name, score + from ${pkTable}@incr( + 'startSnapshotId'='${pkCp0}', + 'endSnapshotId'='${pkCpRenameDelete}' + ) + order by id + """) + sql """set enable_paimon_cpp_reader=true""" + incrementalCpp = sql(""" + select id, full_name, score + from ${pkTable}@incr( + 'startSnapshotId'='${pkCp0}', + 'endSnapshotId'='${pkCpRenameDelete}' + ) + order by id + """) + assertEquals(incrementalJni, incrementalCpp) + + // Scenario TC03/S16: partition pruning and renamed payloads bind to their own snapshots. + assertEquals([[1, "p1", "old"], [2, "p2", "old"]], + sql(""" + select id, old_partition, payload + from ${partitionTable} for version as of ${partitionCp0} + where old_partition = 'p1' or old_partition = 'p2' + order by id + """)) + assertEquals([[1, "p1", "old"], [2, "p2", "old"], [3, "p3", "new"]], + sql(""" + select id, old_partition, new_payload + from ${partitionTable} for version as of ${partitionCpRename} + order by id + """)) + assertUnknownColumn(""" + select new_payload + from ${partitionTable} for version as of ${partitionCp0} + """, "new_payload") + + // Scenario TC08/S20: illegal PK/partition changes fail atomically. + test { + sql """alter table ${pkTable} drop column id""" + exception "Drop column operation is not supported" + } + test { + sql """alter table ${partitionTable} drop column old_partition""" + exception "Drop column operation is not supported" + } + assertEquals([[1, "alpha-updated"], [3, "gamma"]], + sql("""select id, full_name from ${pkTable} order by id""")) + assertEquals([[1, "p1", "old"], [2, "p2", "old"], [3, "p3", "new"]], + sql("""select id, old_partition, new_payload from ${partitionTable} order by id""")) + + // Scenario TC09/R13/R17: cache, JNI and CPP paths return the same historical schema/data. + sql """switch ${noCacheCatalogName}""" + sql """use ${dbName}""" + assertEquals(topCp0Rows, sql(""" + select id, old_name, victim, metric + from ${topTable} for version as of ${topCp0} + order by id + """)) + sql """switch ${catalogName}""" + sql """use ${dbName}""" + sql """set enable_paimon_cpp_reader=false""" + sql """set force_jni_scanner=true""" + List> forcedJniRows = sql(""" + select id, old_name, victim, metric + from ${topTable} for version as of ${topCp0} + order by id + """) + sql """set force_jni_scanner=false""" + sql """set enable_paimon_cpp_reader=true""" + List> cppRows = sql(""" + select id, old_name, victim, metric + from ${topTable} for version as of ${topCp0} + order by id + """) + assertEquals(forcedJniRows, cppRows) + + // Scenario T10/T11: retained tags survive expiration; missing refs never fall back to latest. + spark_paimon """ + ALTER TABLE paimon.${dbName}.${topTable} + SET TBLPROPERTIES ('snapshot.num-retained.min'='1') + """ + spark_paimon """ + CALL paimon.sys.expire_snapshots( + table => '${dbName}.${topTable}', + retain_max => 1 + ) + """ + sql """refresh table ${topTable}""" + assertEquals(topCp0Rows, sql(""" + select id, old_name, victim, metric + from ${topTable} for version as of 'top_cp0' + order by id + """)) + test { + sql """select * from ${topTable} for version as of 9223372036854775807""" + exception "snapshot" + } + test { + sql """select * from ${topTable} for version as of 'missing_schema_tag'""" + exception "tag" + } + } finally { + sql """set enable_paimon_cpp_reader=false""" + sql """set force_jni_scanner=false""" + sql """drop catalog if exists ${catalogName}""" + sql """drop catalog if exists ${noCacheCatalogName}""" + } +}