Handle NULLs in ALTER MODIFY COLUMN by vdimir · Pull Request #84770 · ClickHouse/ClickHouse

vdimir · 2025-07-31T10:13:50Z

Changelog category (leave one):

Backward Incompatible Change

Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):

ALTER MODIFY COLUMN now requires explicit DEFAULT when converting nullable columns to non-nullable types. Previously such ALTERs could get stuck with cannot convert null to not null errors, now NULLs are replaced with column's default expression. Resolves #5985

Details

Close #5985

clickhouse-gh · 2025-07-31T10:14:17Z

Workflow [PR], commit [cd53684]

Summary: ❌

job_name	test_name	status	info
BuzzHouse (amd_msan)		failure
	Logical error: '(isConst() \|\| isSparse() \|\| isReplicated()) ? getDataType() == rhs.getDataType() : typeid(*this) == typeid(rhs)'	FAIL	cidb, issue
Integration tests (amd_asan, db disk, old analyzer, 3/6)		error
	OOM in dmesg	FAIL	cidb

antaljanosbenjamin

LGTM.

den-crane · 2025-07-31T14:29:02Z

Alter of nullable columns to not nullable is not supported #5985

vdimir · 2025-12-15T15:51:36Z

Integration tests (amd_asan, db disk, old analyzer, 3/6) error
OOM in dmesg FAIL cidb

Seems like #84881

bakwc · 2026-01-03T15:21:00Z

Hi team,
just a heads-up that the recent changes appear to have broken backward compatibility.
We spent quite a bit of time investigating an issue on our side before realizing it was related to changes in ClickHouse.
It would be greatly appreciated if the impact on existing integrations and dependent projects could be considered in future updates. Thank you!

antaljanosbenjamin · 2026-01-05T09:22:14Z

just a heads-up that the recent changes appear to have broken backward compatibility.

Hey! Did it broke in a way that is not mentioned in the changelog item? If yes, we have a problem. If not, then it is kind of expected.

Unfortunately from time to time we have to bite the bullet to introduce backward incompatible changes in order to fix big issues, in this case stuck mutations. We try to keep backward incompatible changes minimal and try to always add a proper changelog item, however we might mistakes (not noticing backward incompatibility or missing parts of the scope).

…umn_null_to_default Handle NULLs in ALTER MODIFY COLUMN

25.8.15 Stable backport of ClickHouse#84770: Handle NULLs in ALTER MODIFY COLUMN

…umn_null_to_default Handle NULLs in ALTER MODIFY COLUMN

24.8.14 Backport of ClickHouse#84770: Handle NULLs in ALTER MODIFY COLUMN

…umn_null_to_default Handle NULLs in ALTER MODIFY COLUMN

PR ClickHouse#84770 generates _CAST(ifNull(col, default), TargetType). When source and target base types differ (e.g. Nullable(UInt8) -> String), ifNull throws NO_COMMON_TYPE because getLeastSupertype({UInt8, String}) fails. Fix: swap to ifNull(_CAST(col, Nullable(TargetType)), _CAST(default, TargetType)). Ref: ClickHouse#84770, ClickHouse#5985 Signed-off-by: Daniel Q. Kim <daniel.kim@altinity.com>

PR ClickHouse#84770 generates _CAST(ifNull(col, default), TargetType). When source and target base types differ (e.g. Nullable(UInt8) -> String), ifNull throws NO_COMMON_TYPE because getLeastSupertype({UInt8, String}) fails. Fix: swap to ifNull(_CAST(col, Nullable(TargetType)), _CAST(default, TargetType)). This works independently of use_variant_as_common_type and handles both same-type and cross-type nullable conversions correctly. Signed-off-by: Daniel Q <il9ue@users.noreply.github.com>

…ion test Follow-up to ClickHouse#84770 / the LowCardinality revision. convertRequiredExpressions now branches on whether the target type can be nested in Nullable: - Nullable-able targets (String, LowCardinality(String), numerics): ifNull(_CAST(col, 'makeNullableOrLowCardinalityNullable(T)'), _CAST(default, 'T')) NULLs ride through the cast unparsed, so this path is exception-safe regardless of session settings. - Targets that cannot be nested in Nullable (Array/Tuple/Map/...): if(isNull(col), _CAST(default, 'T'), _CAST(assumeNotNull(col), 'T')) There is no valid Nullable(T) to cast through, so the conversion branches per row. Behavior note: the second form relies on short-circuit evaluation. On a NULL row, assumeNotNull(col) exposes the nested placeholder (e.g. '' for a String source) and _CAST of that placeholder can throw (CANNOT_READ_ARRAY_FROM_TEXT for an Array target). With short_circuit_function_evaluation enabled (the default) the value branch is skipped for NULL rows, so the default is used and no parse is attempted. The regression test pins short_circuit_function_evaluation = 'enable' and tags no-random-settings to keep this deterministic. The first (Nullable-able) path has no such dependency. Test 03575_modify_column_null_to_default_cross_type covers all three target families (scalar String, LowCardinality(String), Array(UInt8)), each with a NULL source row that must take the DEFAULT, and reproduces on upstream/master with NO_COMMON_TYPE when use_variant_as_common_type = 0.

…ion test Follow-up to ClickHouse#84770 / the LowCardinality revision. convertRequiredExpressions now branches on whether the target type can be nested in Nullable: - Nullable-able targets (String, LowCardinality(String), numerics): ifNull(_CAST(col, 'makeNullableOrLowCardinalityNullable(T)'), _CAST(default, 'T')) NULLs ride through the cast unparsed, so this path is exception-safe regardless of session settings. - Targets that cannot be nested in Nullable (Array/Tuple/Map/...): _CAST(ifNull(col, _CAST(default, 'X')), 'T') (X = source base type) NULL rows are replaced by the default re-encoded in the source type before any cast to T, so the placeholder is never parsed. There is no branch to short-circuit, so this path is exception-safe regardless of session settings. Test 03575_modify_column_null_to_default_cross_type covers all three target families (scalar String, LowCardinality(String), Array(UInt8)), each with a NULL source row that must take the DEFAULT, plus an Array case run with short_circuit_function_evaluation = 'disable' to prove the conversion does not depend on the setting. It reproduces on upstream/master with NO_COMMON_TYPE when use_variant_as_common_type = 0. Signed-off-by: Daniel Q <il9ue@users.noreply.github.com>

…column type Follow-up to ClickHouse#84770 / the LowCardinality revision. convertRequiredExpressions builds the null-filling conversion and now pins the result to exactly required_column.type on both branches: - Nullable-able targets (String, LowCardinality, numerics): _CAST(ifNull(_CAST(col, 'LowCardinality(Nullable(T))'), _CAST(default, 'T')), 'T') The inner ifNull already resolves to the non-nullable target because the default arg is non-nullable; the outer _CAST pins the type explicitly so the generated column does not depend on ifNull's supertype resolution, and keeps this branch symmetric with the Array branch below. - Targets that cannot be nested in Nullable (Array/Tuple/Map/...): _CAST(ifNull(col, _CAST(default, 'X')), 'T') (X = source base type) NULL rows are replaced by the default re-encoded in the source type before any cast to T, so the placeholder is never parsed; no branch to short-circuit, so independent of short_circuit_function_evaluation. Test 03575 covers scalar String, LowCardinality(String), and Array(UInt8) targets, each with a NULL source row taking the DEFAULT, materialized with OPTIMIZE TABLE ... FINAL to exercise the part-writer path, plus an Array case under short_circuit_function_evaluation = 'disable'. Reproduces on upstream/master with NO_COMMON_TYPE when use_variant_as_common_type = 0. Signed-off-by: Daniel Q <il9ue@users.noreply.github.com>

…ple/Map) Follow-up to ClickHouse#84770 / the LowCardinality revision. `convertRequiredExpressions` builds the null-filling conversion and pins the result to exactly `required_column.type` on both branches: - Nullable-able targets (`String`, `LowCardinality`, numerics): `_CAST(ifNull(_CAST(col, 'LowCardinality(Nullable(T))'), _CAST(default, 'T')), 'T')` The inner `ifNull` already resolves to the non-nullable target because its default arg is non-nullable; the outer `_CAST` pins the type explicitly so the generated column does not depend on `ifNull`'s supertype resolution, and keeps this branch symmetric with the else branch below. - Targets that cannot be nested in `Nullable` (`Array`/`Tuple`/`Map`/...): `if(isNull(col), _CAST(default, 'T'), _CAST(assumeNotNull(col), 'T'))` The default now stays on the TARGET side. The previous construction re-encoded the default in the source base type, `_CAST(ifNull(col, _CAST(default, 'X')), 'T')` (X = source base type), which threw `TYPE_MISMATCH` whenever the target default is not castable back to the source type. For example, a `Map` target with a `map(...)` default over a `Nullable(Tuple(...))` source built `_CAST(map('a', 1), 'Tuple(...)')`, and `CAST AS Tuple` rejects `Map` input. Keeping the default in the target type avoids the source round-trip entirely. The value branch `_CAST(assumeNotNull(col), 'T')` would throw on NULL rows if it parsed the `assumeNotNull` placeholder, but `performRequiredConversions` runs the expression through `ExpressionActions` that lazily evaluate short-circuit functions regardless of `short_circuit_function_evaluation`, so the branch is only evaluated for non-NULL rows. Measured: a plain top-level `if` over the same placeholder throws under `short_circuit_function_evaluation = 'disable'`, while this conversion path does not. Test 03575 covers scalar `String`, `LowCardinality(String)`, `Array(UInt8)`, `Tuple`, and `Map` targets, each with a NULL source row taking the `DEFAULT`, materialized with `OPTIMIZE TABLE ... FINAL` to exercise the part-writer path, plus an `Array` case under `short_circuit_function_evaluation = 'disable'`. The scalar cases reproduce on upstream/master with `NO_COMMON_TYPE` when `use_variant_as_common_type = 0`; the new `Map`-over-`Tuple` regression (Case 9) reproduces with `TYPE_MISMATCH` on the previous source-round-trip else branch. Signed-off-by: Daniel Q <il9ue@users.noreply.github.com>

clickhouse-gh Bot added the pr-backward-incompatible Pull request with backwards incompatible changes label Jul 31, 2025

antaljanosbenjamin self-assigned this Jul 31, 2025

antaljanosbenjamin approved these changes Jul 31, 2025

View reviewed changes

vdimir force-pushed the vdimir/modify_column_null_to_default branch from 23a2bdd to ff270c5 Compare August 18, 2025 07:31

vdimir force-pushed the vdimir/modify_column_null_to_default branch from f3140f1 to 238dcbb Compare September 3, 2025 13:10

vdimir enabled auto-merge September 3, 2025 13:11

Handle NULLs in ALTER MODIFY COLUMN

6ab2401

vdimir force-pushed the vdimir/modify_column_null_to_default branch from 6b5c46e to 6ab2401 Compare December 11, 2025 14:07

ClickHouse deleted a comment from clickhouse-gh Bot Dec 11, 2025

update 03721_statistics_alter_type_bug.sql

cd53684

vdimir added this pull request to the merge queue Dec 15, 2025

Merged via the queue into master with commit 58d29ac Dec 15, 2025
127 of 130 checks passed

vdimir deleted the vdimir/modify_column_null_to_default branch December 15, 2025 16:04

robot-ch-test-poll added the pr-synced-to-cloud The PR is synced to the cloud repo label Dec 15, 2025

bakwc mentioned this pull request Jan 3, 2026

feat #67: support clickhouse cluster & distributed tables bakwc/mysql_ch_replicator#223

Merged

zvonand pushed a commit to Altinity/ClickHouse that referenced this pull request Jan 15, 2026

Merge pull request ClickHouse#84770 from ClickHouse/vdimir/modify_col…

c02739b

…umn_null_to_default Handle NULLs in ALTER MODIFY COLUMN

zvonand mentioned this pull request Jan 15, 2026

25.8.14 Stable backport of #84770: Handle NULLs in ALTER MODIFY COLUMN Altinity/ClickHouse#1314

Closed

25 tasks

zvonand pushed a commit to Altinity/ClickHouse that referenced this pull request Jan 27, 2026

Merge pull request ClickHouse#84770 from ClickHouse/vdimir/modify_col…

08f97b7

…umn_null_to_default Handle NULLs in ALTER MODIFY COLUMN

zvonand mentioned this pull request Jan 27, 2026

25.8.15 Stable backport of #84770: Handle NULLs in ALTER MODIFY COLUMN Altinity/ClickHouse#1344

Merged

25 tasks

zvonand added a commit to Altinity/ClickHouse that referenced this pull request Jan 30, 2026

Merge pull request #1344 from Altinity/backports/25.8.15/84770

89bfe4f

25.8.15 Stable backport of ClickHouse#84770: Handle NULLs in ALTER MODIFY COLUMN

zvonand pushed a commit to Altinity/ClickHouse that referenced this pull request Feb 5, 2026

Merge pull request ClickHouse#84770 from ClickHouse/vdimir/modify_col…

2165a97

…umn_null_to_default Handle NULLs in ALTER MODIFY COLUMN

zvonand mentioned this pull request Feb 5, 2026

24.8.14 Backport of #84770: Handle NULLs in ALTER MODIFY COLUMN Altinity/ClickHouse#1370

Merged

25 tasks

zvonand added a commit to Altinity/ClickHouse that referenced this pull request Feb 7, 2026

Merge pull request #1370 from Altinity/backports/24.8.14/84770

b63d329

24.8.14 Backport of ClickHouse#84770: Handle NULLs in ALTER MODIFY COLUMN

zvonand pushed a commit to Altinity/ClickHouse that referenced this pull request Mar 17, 2026

Merge pull request ClickHouse#84770 from ClickHouse/vdimir/modify_col…

81628aa

…umn_null_to_default Handle NULLs in ALTER MODIFY COLUMN

zvonand mentioned this pull request Mar 17, 2026

24.8.14 Stable Pre-Release PR Altinity/ClickHouse#1540

Merged

il9ue mentioned this pull request Apr 1, 2026

24.8 Fix for #84770 backport - Fix Nullable(X)->Y cross-type ALTER MODIFY COLUMN Altinity/ClickHouse#1611

Open

25 tasks

il9ue mentioned this pull request Apr 1, 2026

25.8 Fix for #84770 backport - Fix Nullable(X)->Y cross-type ALTER MODIFY COLUMN Altinity/ClickHouse#1612

Open

25 tasks

il9ue mentioned this pull request Apr 9, 2026

Fix Nullable(X)->Y cross-type ALTER MODIFY COLUMN #102156

Merged

mkmkme mentioned this pull request Aug 5, 2026

Antalya 25.8 backport of #84770: Handle NULLs in ALTER MODIFY COLUMN Altinity/ClickHouse#2169

Open

28 tasks

Navigation Menu

Uh oh!

Handle NULLs in ALTER MODIFY COLUMN - #84770

Handle NULLs in ALTER MODIFY COLUMN#84770
vdimir merged 2 commits into
masterfrom
vdimir/modify_column_null_to_default

vdimir commented Jul 31, 2025 •

edited by clickhouse-gh Bot

Loading

Uh oh!

clickhouse-gh Bot commented Jul 31, 2025 •

edited

Loading

Uh oh!

antaljanosbenjamin left a comment

Uh oh!

den-crane commented Jul 31, 2025

Uh oh!

vdimir commented Dec 15, 2025

Uh oh!

Uh oh!

bakwc commented Jan 3, 2026

Uh oh!

antaljanosbenjamin commented Jan 5, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Uh oh!

Conversation

vdimir commented Jul 31, 2025 • edited by clickhouse-gh Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changelog category (leave one):

Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):

Details

Uh oh!

clickhouse-gh Bot commented Jul 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

antaljanosbenjamin left a comment

Choose a reason for hiding this comment

Uh oh!

den-crane commented Jul 31, 2025

Uh oh!

vdimir commented Dec 15, 2025

Uh oh!

Uh oh!

bakwc commented Jan 3, 2026

Uh oh!

antaljanosbenjamin commented Jan 5, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

vdimir commented Jul 31, 2025 •

edited by clickhouse-gh Bot

Loading

clickhouse-gh Bot commented Jul 31, 2025 •

edited

Loading