Antalya 26.6: Partition export + cluster functions

[zvonand]

Changelog category (leave one):

• New Feature

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Add support for exporting MergeTree parts and partitions to object storage, including experimental exports to Apache Iceberg, with a command to schedule exports for all partitions at once, automatic type casting as in INSERT ... SELECT, retries with backoff for transient failures, and a persistent history of export tasks in system.replicated_partition_exports. Add alternative syntax for cluster table functions with distributed execution of JOINs on swarm nodes, and speed up object storage queries through caching of S3 ListObjects calls and pre-aggregation on remote nodes.
Add support for exporting MergeTree parts and partitions to object storage, including experimental exports to Apache Iceberg, with a command to schedule exports for all partitions at once, automatic type casting as in INSERT ... SELECT, retries with backoff for transient failures, and a persistent history of export tasks in system.replicated_partition_exports. Add alternative syntax for cluster table functions with distributed execution of JOINs on swarm nodes, and speed up object storage queries through caching of S3 ListObjects calls and pre-aggregation on remote nodes (#1718 by @zvonand, #1646 by @ianton-ru, #1640 by @ianton-ru, #1744 by @zvonand, #1713 by @arthurpassos, #1783 by @ianton-ru, #1741 by @arthurpassos, #1728 by @arthurpassos, #1730 by @arthurpassos, #1813 by @arthurpassos, #1836 by @arthurpassos, #1847 by @arthurpassos, #1782 by @ianton-ru, #1845 by @ianton-ru, #1856 by @ianton-ru, #1863 by @ianton-ru, #1872 by @ianton-ru, #1917 by @arthurpassos, #1779 by @arthurpassos, #1912 by @arthurpassos, #1990 by @arthurpassos, #2004 by @arthurpassos, #1984 by @arthurpassos).

CI/CD Options

Exclude tests:

• Fast test
• Integration Tests
• Stateless tests
• Stateful tests
• Performance tests
• All with ASAN
• All with TSAN
• All with MSAN
• All with UBSAN
• All with Coverage
• All with Aarch64
• All Regression
• Disable CI Cache

Regression jobs to run:

• Fast suites (mostly <1h)
• Aggregate Functions (2h)
• Alter (1.5h)
• Benchmark (30m)
• ClickHouse Keeper (1h)
• Iceberg (2h)
• LDAP (1h)
• Parquet (1.5h)
• RBAC (1.5h)
• SSL Server (1h)
• S3 (2h)
• S3 Export (2h)
• Swarms (30m)
• Tiered Storage (2h)

Combined port of 23 PR(s) (group auto-grp-pr-1718). Cherry-picked from #1718, #1646, #1640, #1744, #1713, #1783, #1741, #1728, #1730, #1813, #1836, #1847, #1782, #1845, #1856, #1863, #1872, #1917, #1779, #1912, #1990, #2004, #1984.

• Antalya 26.3: apassos-3: combined port of 12 PRs #1718 — Antalya 26.3: apassos-3: combined port of 12 PRs
• 26.3 Antalya port - fixes for s3Cluster distributed calls #1646 — 26.3 Antalya port - fixes for s3Cluster distributed calls
• 26.3 Antalya port - Alternative syntax for cluster functions #1640 — 26.3 Antalya port - Alternative syntax for cluster functions
• Antalya 26.3: Fix export task not being killed during s3 outage #1744 — Antalya 26.3: Fix export task not being killed during s3 outage
• Remove lock inside the task export partition #1713 — Remove lock inside the task export partition
• Fix incorrect interaction between object_storage_remote_initiator and object_storage_cluster. #1783 — Fix incorrect interaction between object_storage_remote_initiator and object_storage_cluster.
• Export partition all #1741 — Export partition all
• Make export part and partition respect background tasks memory limit #1728 — Make export part and partition respect background tasks memory limit
• improve observability over exceptions #1730 — improve observability over exceptions
• dont commit while holding the lock on cleanup thread #1813 — dont commit while holding the lock on cleanup thread
• Do not require iceberg insert settings on replicas #1836 — Do not require iceberg insert settings on replicas
• Export partition - Introduce commit lock #1847 — Export partition - Introduce commit lock
• Antalya 26.3: Cluster Joins part 2 - global mode #1782 — Antalya 26.3: Cluster Joins part 2 - global mode
• Do not make 'describe table' query when schema is known #1845 — Do not make 'describe table' query when schema is known
• Fix test_remote_initiator_after_non_remote #1856 — Fix test_remote_initiator_after_non_remote
• Fix cluster functions with hive partitioning #1863 — Fix cluster functions with hive partitioning
• Fix aggregation flow with remote initiator #1872 — Fix aggregation flow with remote initiator
• Turn system.replicated_partition_exports table into a history table by removing the TTL #1917 — Turn system.replicated_partition_exports table into a history table by removing the TTL
• Auto cast values on export part and partition just like insert select #1779 — Auto cast values on export part and partition just like insert select
• Replace the export partition local manifests lock by a multiversion container #1912 — Replace the export partition local manifests lock by a multiversion container
• Preserve a few settings to partition export #1990 — Preserve a few settings to partition export
• Change export partition log levels #2004 — Change export partition log levels
• Partition export per part local backoff policy #1984 — Partition export per part local backoff policy

[github-actions]

Workflow [PR], commit [4b71fda]

[zvonand]

RelEasy analyze-fails — DONE

run completed at 2026-08-12T13:59:37Z

• Head SHA: ea0816eea1 (feature/antalya-26.6/auto-grp-pr-1718)
• Tests considered: 2 across 1 CI shard(s)
• Outcomes: 1 done · 0 partial · 0 unrelated · 0 unresolved
• Commits added by AI: 1 (✅ pushed)
• Anthropic cost: $4.0495

Per-shard outcomes

✅ DONE — Fast test

2 failed test(s) considered — +1 commit(s) — cost $4.0495
full report

AI narration

I'll start by inspecting the repository state and the PR diff.
Now let me apply both fixes.
Now the settings history entries:

Summary

Triage (Step 1). The PR is a group merge that pulls a large Antalya feature set into antalya-26.6. Both failures are traceable to the merged content:

Root cause A — 02995_new_settings_history: the merge brought in three new settings (lock_object_storage_task_distribution_ms, allow_experimental_iceberg_read_optimization, allow_retries_in_cluster_requests) whose SettingsChangesHistory.cpp entries were left commented out during conflict resolution (the head commit ea0816eea1f un-commented some neighbours but not these). Verified: base branch has neither the settings nor the entries; HEAD has the settings but not the entries.

Root cause B — 04401_system_reset_ddl_worker_access: the merge combined the backport that adds ASTSystemQuery::Type::RESET_DDL_WORKER (last enumerator) with Antalya-only SYSTEM types (STOP/START_SWARM_MODE, DROP_OBJECT_STORAGE_LIST_OBJECTS_CACHE, …). The enum grew from 126 values on antalya-26.6 to 129 on HEAD, so RESET_DDL_WORKER == 128 fell outside magic_enum's default range [-128, 127]. ParserSystemQuery enumerates types via magic_enum::enum_values, so SYSTEM RESET DDL WORKER stopped parsing entirely → no ACCESS_DENIED (and no "grant SYSTEM RESET DDL WORKER" message on the ON CLUSTER path), while CHECK GRANT (which doesn't go through that parser) still reported the privilege — exactly the observed 2-line diff. ASTSystemQuery::typeToString also indexed a magic_enum-sized array out of bounds.

Fix (Step 3, one build). src/Parsers/ASTSystemQuery.h: specialize magic_enum::customize::enum_range<DB::ASTSystemQuery::Type> (0..512) + include base/EnumReflection.h. src/Core/SettingsChangesHistory.cpp: un-comment the three Antalya setting entries. bash .releasy/build.sh succeeded.

Verification (Step 4). The pre-existing local binary was stale (lacked the merged settings), so results only became meaningful after the rebuild. Batch re-run of both failures plus two access-related regression spot-checks:

[1 / 4] 01271_show_privileges:                 [ OK ]
[2 / 4] 02995_new_settings_history:            [ OK ]
[3 / 4] 04401_system_reset_ddl_worker_access:  [ OK ]
[4 / 4] 03247_show_grants_with_implicit:       [ OK ]

Committed as 4b71fda9f2e (append-only); git status --porcelain is empty.

Per-test outcome

• 04401_system_reset_ddl_worker_access — [fixed] — this PR's merge pushed ASTSystemQuery::Type past magic_enum's default 127 limit so SYSTEM RESET DDL WORKER no longer parsed; range extended, test passes.
• 02995_new_settings_history — [fixed] — three settings added by this PR's merge had their SettingsChangesHistory.cpp entries left commented out; un-commented, test passes.

DONE

Summary

Triage (Step 1). The PR is a group merge that pulls a large Antalya feature set into antalya-26.6. Both failures are traceable to the merged content:

Root cause A — 02995_new_settings_history: the merge brought in three new settings (lock_object_storage_task_distribution_ms, allow_experimental_iceberg_read_optimization, allow_retries_in_cluster_requests) whose SettingsChangesHistory.cpp entries were left commented out during conflict resolution (the head commit ea0816eea1f un-commented some neighbours but not these). Verified: base branch has neither the settings nor the entries; HEAD has the settings but not the entries.

Root cause B — 04401_system_reset_ddl_worker_access: the merge combined the backport that adds ASTSystemQuery::Type::RESET_DDL_WORKER (last enumerator) with Antalya-only SYSTEM types (STOP/START_SWARM_MODE, DROP_OBJECT_STORAGE_LIST_OBJECTS_CACHE, …). The enum grew from 126 values on antalya-26.6 to 129 on HEAD, so RESET_DDL_WORKER == 128 fell outside magic_enum's default range [-128, 127]. ParserSystemQuery enumerates types via magic_enum::enum_values, so SYSTEM RESET DDL WORKER stopped parsing entirely → no ACCESS_DENIED (and no "grant SYSTEM RESET DDL WORKER" message on the ON CLUSTER path), while CHECK GRANT (which doesn't go through that parser) still reported the privilege — exactly the observed 2-line diff. ASTSystemQuery::typeToString also indexed a magic_enum-sized array out of bounds.

Fix (Step 3, one build). src/Parsers/ASTSystemQuery.h: specialize magic_enum::customize::enum_range<DB::ASTSystemQuery::Type> (0..512) + include base/EnumReflection.h. src/Core/SettingsChangesHistory.cpp: un-comment the three Antalya setting entries. bash .releasy/build.sh succeeded.

Verification (Step 4). The pre-existing local binary was stale (lacked the merged settings), so results only became meaningful after the rebuild. Batch re-run of both failures plus two access-related regression spot-checks:

[1 / 4] 01271_show_privileges:                 [ OK ]
[2 / 4] 02995_new_settings_history:            [ OK ]
[3 / 4] 04401_system_reset_ddl_worker_access:  [ OK ]
[4 / 4] 03247_show_grants_with_implicit:       [ OK ]

Committed as 4b71fda9f2e (append-only); git status --porcelain is empty.

Per-test outcome

• 04401_system_reset_ddl_worker_access — [fixed] — this PR's merge pushed ASTSystemQuery::Type past magic_enum's default 127 limit so SYSTEM RESET DDL WORKER no longer parsed; range extended, test passes.
• 02995_new_settings_history — [fixed] — three settings added by this PR's merge had their SettingsChangesHistory.cpp entries left commented out; un-commented, test passes.

DONE

---

🤖 Posted automatically by releasy analyze-fails. Re-run the command to refresh.