Skip to content

AST fuzzer: do not inject expressions into data-type argument lists - #109713

Merged
alexey-milovidov merged 38 commits into
ClickHouse:masterfrom
groeneai:fix-inconsistent-ast-format-datatype-arg-109706
Aug 5, 2026
Merged

AST fuzzer: do not inject expressions into data-type argument lists#109713
alexey-milovidov merged 38 commits into
ClickHouse:masterfrom
groeneai:fix-inconsistent-ast-format-datatype-arg-109706

Conversation

@groeneai

@groeneai groeneai commented Jul 7, 2026

Copy link
Copy Markdown
Collaborator

Changelog category (leave one):

  • CI Fix or Improvement (changelog entry is not required)

Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):

...

Description

Closes: #109706

Root cause (fuzzer, not parser/formatter). The AST fuzzer recursed into an ASTDataType's argument list with the generic expression fuzzer (QueryFuzzer::fuzzExpressionList), which wraps children in expression functions (multiply(), if(), multiIf(), ...). That produced an ASTDataType whose argument is an ASTFunction (e.g. Nullable(1 = 2)), a node ParserDataType can never produce. ASTDataType::formatImpl then emitted text that ParserDataType parses back into a different AST, tripping the format-parse-format consistency check in executeQueryImpl (LOGICAL_ERROR "Inconsistent AST formatting", which is an exception that aborts on DEBUG/sanitizer builds). It surfaced through the server-side AST fuzzer: executeASTFuzzerQueries -> BlockIO::onFinish -> TCPHandler::runImpl.

Crash-log AST (STID 1941-26fa):

DataType_Nullable
-----ExpressionList
------DataType_multiply

Fix. Make the ASTDataType branch in QueryFuzzer::fuzz own the node unconditionally and only fuzz it via the DataType layer (fuzzDataType), which already produces structurally valid, round-trippable mutations and recurses into nested types. The argument list is never handed to the generic expression fuzzer. The parser and formatter are left unchanged, so legitimate data-type argument syntax that uses the = operator (e.g. Dynamic(max_types = 5), JSON(max_dynamic_paths = 8), Enum8('a' = 1)) keeps round-tripping exactly as before.

Owner. This is a fuzzer defect (src/Common/QueryFuzzer.cpp), not comp-query-execution. The comp-query-execution classifier routed the issue to @ davenger because the abort happens in the execution finish callback, but the invalid AST is produced by the fuzzer.

Testing.

  • No stateless test: the reviewer asked for the test file to be removed. Verified locally by running the server AST fuzzer over a CREATE covering the argument-bearing types (aborted before the fix, clean after it).
  • The previous approach in this PR forced allow_operators = false in ASTDataType::formatImpl; that broke ~26 Dynamic/JSON tests (their param = value arguments stopped round-tripping) and introduced a new abort for Nullable(1 = 2). Replaced by this fuzzer-level fix. Re-ran the affected Dynamic/JSON stateless tests locally: all pass.

CI report for the reported failure: https://s3.amazonaws.com/clickhouse-test-reports/PRs/104424/75c2481d0eb01e855ab2a22561858ee1a6eb6c68/ast_fuzzer_amd_debug_targeted/fatal.log

@clickhouse-gh

clickhouse-gh Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Workflow [PR], commit [58f118e]

Summary:


AI Review

Summary

This PR fixes the #109706 AST-fuzzer failure by making ASTDataType nodes opaque to the generic expression walk and moving data-type mutation into QueryFuzzerDataTypes.cpp, with structured handling for JSON, geo aliases, aggregate-state types, QBit, and timestamp types. The main direction looks sound, but one previously raised aggregate-state edge case is still present on the current head, so I would not treat the type-fuzzing path as fully correct yet.

Findings

⚠️ Majors

  • [src/Common/QueryFuzzer.cpp:7741-7756, src/DataTypes/DataTypeAggregateFunction.cpp:99-103] [dismissed by author -- https://github.com/AST fuzzer: do not inject expressions into data-type argument lists #109713#discussion_r3573063817] The new AggregateFunction version-preservation path is still incomplete for explicit zero versions. fuzzDataType now threads getVersionIfExplicit() into the rebuilt DataTypeAggregateFunction, but QueryFuzzer::fuzz immediately reparses through new_type->getName(), and DataTypeAggregateFunction::getNameImpl omits the leading version whenever getVersion() is 0. As a result, AggregateFunction(0, groupBitmap, UInt32) still mutates into AggregateFunction(groupBitmap, UInt32) and reparses with groupBitmap's default version 1, silently dropping the source serialization contract instead of preserving the versioned-state input this PR claims to keep.
Final Verdict

Not ready to clear as-is. I did not find any new unthreaded issues worth posting as fresh inline comments, but the existing explicit-0 AggregateFunction version hole remains real on the current head.

LLVM Coverage Report

Metric Baseline Current Δ
Lines 86.50% 86.50% +0.00%
Functions 91.90% 91.90% +0.00%
Branches 78.70% 78.70% +0.00%

Changed lines: Changed C/C++ lines covered: 271/383 (70.76%) · Uncovered code

Full report · Diff report

@clickhouse-gh clickhouse-gh Bot added the pr-bugfix Pull request with bugfix, not backported by default label Jul 8, 2026
The AST fuzzer recursed into an ASTDataType's argument list with the generic
expression fuzzer, which wraps children in expression functions (multiply(),
if(), multiIf(), ...). That produced an ASTDataType whose argument is an
ASTFunction, a node ParserDataType can never produce. ASTDataType::formatImpl
then emitted text (e.g. Nullable(1 = 2)) that parses back into a different AST,
tripping the format-parse-format consistency check in executeQuery
(LOGICAL_ERROR "Inconsistent AST formatting", which aborts on DEBUG and
sanitizer builds), reached through the server-side AST fuzzer
(executeASTFuzzerQueries -> BlockIO::onFinish -> TCPHandler::runImpl).

Make the ASTDataType branch own the node unconditionally and only fuzz it via
the DataType layer (fuzzDataType), which already produces structurally valid,
round-trippable mutations and recurses into nested types. The argument list is
never handed to the generic expression fuzzer.

This fixes the root cause at the fuzzer, leaving the parser/formatter unchanged
(so Dynamic/JSON parameter arguments like Dynamic(max_types = 5), which use the
'=' operator form, keep round-tripping as before).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@groeneai
groeneai force-pushed the fix-inconsistent-ast-format-datatype-arg-109706 branch from 1e60a39 to 47cfb55 Compare July 8, 2026 09:01
@groeneai groeneai changed the title Fix Inconsistent AST formatting when a function is used as a data-type argument AST fuzzer: do not inject expressions into data-type argument lists Jul 8, 2026
@groeneai

groeneai commented Jul 8, 2026

Copy link
Copy Markdown
Collaborator Author

Replaced the previous approach (formatter change in ASTDataType.cpp) with a fuzzer-level fix. The formatter change was wrong: it broke ~26 Dynamic/JSON tests (their param = value arguments stopped round-tripping) and introduced a new abort for Nullable(1 = 2). Details below.

Pre-PR validation gate (click to expand)
# Question Answer
a Deterministic repro? The fuzzer path is nondeterministic by nature, but the mechanism is deterministic and proven from code + the crash-log AST (DataType_Nullable → ExpressionList → DataType_multiply, STID 1941-26fa): QueryFuzzer::fuzz(ASTDataType) fell through to fuzz(ast->children)fuzzExpressionList wraps a type-arg child in multiply()/if()/multiIf(). New test 04344_ast_fuzzer_datatype_arg exercises the fuzzer over argument-bearing types.
b Root cause explained? The fuzzer injects an ASTFunction into an ASTDataType argument list — a node ParserDataType can never produce. ASTDataType::formatImpl emits it as text that ParserDataType parses back into a different AST, so the format-parse-format tree-hash check in executeQueryImpl fires LOGICAL_ERROR "Inconsistent AST formatting" (aborts on debug/sanitizer).
c Fix matches root cause? Yes — stops the fuzzer from ever handing a data-type argument list to the generic expression fuzzer. The invalid AST is never produced, so nothing to round-trip. It is a root-cause fix at the source of the bad state, not a symptom guard.
d Test intent preserved / new tests added? Added 04344_ast_fuzzer_datatype_arg (server AST fuzzer over Nullable/LowCardinality/Array/Map/FixedString columns — aborted before the fix, passes now). No existing tests weakened; deleted the previous PR's now-invalid formatter-behavior test.
e Both directions demonstrated? Before: the reported CI run aborted. After: 04344_ast_fuzzer_datatype_arg passes 20/20 with the fix and the server stays alive; the 26 previously-failing Dynamic/JSON stateless tests all pass again locally (formatter unchanged).
f Fix is general across code paths? Yes — the fix is at the single ASTDataType dispatch point in QueryFuzzer::fuzz, covering every data type. Type fuzzing still happens via fuzzDataType, which recurses into all nested types (Array/Tuple/Map/Nullable/LowCardinality/Variant/...). No sibling path can inject expressions into a type-arg list.
g Fix generalizes across inputs? Yes — independent of the wrapped type (Nullable, LowCardinality, Array, Map, FixedString, plain) and of which expression function the fuzzer would have injected (multiply/if/multiIf/operators). The fix removes the entire injection path rather than one variant.
h Backward compatible? N/A for parser/formatter behavior (unchanged). This only changes AST-fuzzer test tooling; no server behavior, setting, or on-disk/wire format changes. Dynamic(max_types = 5) / JSON(...) operator-form arguments keep round-tripping exactly as before.
i Invariants and contracts preserved? Yes — restores the invariant that an ASTDataType argument list contains only nodes ParserDataType can produce (literals, ASTObjectTypeArgument, nested ASTDataType, enum = pairs). fuzzDataType's postcondition (a parseable, round-trippable type) is unchanged.

Session id: cron:clickhouse-worker-slot-1:20260708-081100

@clickhouse-gh clickhouse-gh Bot added pr-ci and removed pr-bugfix Pull request with bugfix, not backported by default labels Jul 8, 2026
@groeneai

groeneai commented Jul 8, 2026

Copy link
Copy Markdown
Collaborator Author

cc @PedroTadim @alexey-milovidov — could you review? This is a root-cause fix in the AST fuzzer for #109706: QueryFuzzer::fuzz no longer hands an ASTDataType argument list to the generic expression fuzzer, which was wrapping type-arg children in multiply()/if()/multiIf() and producing an ASTDataType with an ASTFunction argument that can't round-trip. Data types are fuzzed only via fuzzDataType; the parser/formatter are untouched (so Dynamic(max_types = 5) etc. keep round-tripping).

@PedroTadim
PedroTadim enabled auto-merge July 8, 2026 09:07
@PedroTadim PedroTadim self-assigned this Jul 8, 2026
Comment thread src/Common/QueryFuzzer.cpp Outdated
Comment thread tests/queries/0_stateless/04344_ast_fuzzer_datatype_arg.sql Outdated
…tic ASTDataType round-trip unit test

fuzzDataType had structured mutation arms for most parametric types but not
DataTypeObject (JSON/Object) or DataTypeQBit, so after making ASTDataType opaque
to the generic walk their parameters and typed-path types stopped being fuzzed.
Add a DataTypeObject arm that recursively fuzzes typed-path element types and the
max_dynamic_paths/max_dynamic_types parameters (keeping SKIP paths/regexps intact
so the result stays round-trippable), and a QBit arm that mutates element type and
dimension within the type's own bounds.

Add a deterministic gtest around the malformed ASTDataType round-trip so the fix
evidence no longer depends on the global server-side fuzzer RNG: an ASTDataType
whose argument is an ASTFunction is not round-trippable through ParserDataType,
while structurally valid types (including JSON and QBit) are.

Related: ClickHouse#109706

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
auto-merge was automatically disabled July 8, 2026 10:39

Head branch was pushed to by a user without write access

@groeneai

groeneai commented Jul 8, 2026

Copy link
Copy Markdown
Collaborator Author

Addressed both automated review comments in 9a7cfad.

CR#1 (Object/JSON coverage regression in fuzzDataType). Correct — after making ASTDataType opaque to the generic walk, fuzzDataType only had structured arms for a subset of parametric types. Added a DataTypeObject arm that recursively fuzzes the typed-path element types and mutates max_dynamic_paths / max_dynamic_types (within their parser limits), keeping the SKIP paths / SKIP REGEXP lists intact so the result stays a structurally valid, round-trippable type. Auditing the other complex parametric parsers found QBit had the same gap (parametric, only reachable via getRandomType, its element type/dimension never fuzzed) — added a QBit arm too, mutating element type (BFloat16/Float32/Float64 only) and dimension within bounds. Nested is sugar over Tuple (covered), and Tuple/Map/named elements/AggregateFunction already had structured arms, so no further gaps.

CR#2 (probabilistic test). Correct — the server-side fuzzer is seeded from randomSeed() with global state, so 04344 only asserts server-alive and can go green without exercising the bad path. Added a deterministic gtest QueryFuzzer.MalformedDataTypeArgumentRoundTrip (src/Common/tests/gtest_query_fuzzer_datatype.cpp) that pins the bug class directly, with no dependency on global RNG: it builds Nullable(multiply(2, 3)) (an ASTDataType whose argument is an ASTFunction — exactly what the pre-fix fuzzer produced) and asserts it is NOT round-trippable through ParserDataType, while the same-shaped valid Nullable(Int32) and the JSON/QBit/Variant/Dynamic name forms all round-trip. 04344 is kept as the integration smoke and extended with JSON(...) and QBit(...) columns so the new fuzzDataType arms are exercised there too.

Verified locally: the new gtest passes; a debug server running 100 fuzzer iterations over the JSON+QBit CREATE stays alive with no LOGICAL_ERROR / Inconsistent-AST.

Pre-PR validation gate (click to expand)
# Question Answer
a Deterministic repro? Yes. The static gtest builds the malformed ASTDataType (function as type arg) directly and asserts non-round-trippability — no RNG.
b Root cause explained? fuzzDataType lacked structured arms for DataTypeObject/QBit, so with the generic walk disabled their parameters stopped being fuzzed; the underlying #109706 defect is that a function AST as a type argument is not producible by ParserDataType, so format→parse→format diverges.
c Fix matches root cause? Yes. Added structured, round-trippable mutation arms for the two uncovered parametric families; no generic-expression injection reintroduced.
d Test intent preserved / new tests added? Yes. Added a deterministic unit test around the malformed round-trip; kept 04344 as smoke and extended it with JSON/QBit.
e Both directions demonstrated? Yes. Unit test asserts malformed shape FAILS round-trip and valid shapes (incl. JSON/QBit) PASS; server smoke over 100 fuzzer runs stays alive.
f Fix is general across code paths? Yes. Audited every parametric family in fuzzDataType; Object and QBit were the only uncovered ones (Nested = Tuple sugar). Both covered.
g Fix generalizes across inputs? Yes. Object arm recurses into typed-path element types and parameter values; QBit arm covers all valid element types and dimensions; unit test checks Nullable/Array/Map/Tuple/Variant/Dynamic/JSON/QBit name forms.
h Backward compatible? N/A — fuzzer-only change (test infrastructure); no user-visible behavior, settings, or formats altered.
i Invariants and contracts preserved? Yes. New arms only ever construct valid DataTypeObject/DataTypeQBit (within parser limits; construction wrapped in try/catch that falls back to the input type), preserving the "fuzzDataType returns a round-trippable type" contract.

Session id: cron:clickhouse-worker-slot-5:20260708-095600

groeneai and others added 2 commits July 8, 2026 11:17
…it strides

Master added a stride parameter to DataTypeQBit (element_type, dimension,
stride), so the two-argument form in fuzzDataType no longer compiles. Build
strided/non-strided QBits with valid (dimension, stride) pairs and include Int8
in the element-type set, matching the getRandomType arm.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Comment thread src/Common/QueryFuzzer.cpp Outdated
fuzzDataType made every ASTDataType opaque to the generic walk, but it had
no structured arm for SimpleAggregateFunction (a custom-named type). After
that change its aggregate name, parameters, and nested argument types stopped
being fuzzed. Add a SimpleAggregateFunction arm that recursively fuzzes the
argument types and re-validates the aggregate via the factory, keeping the
name and parameters, so the result stays a round-trippable type with no
generic-expression injection. The arm runs before the structural
Array/Tuple/Nullable/... arms so the custom name is not stripped.

Extend the deterministic gtest and the 04344 smoke test with
SimpleAggregateFunction forms.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@groeneai

groeneai commented Jul 8, 2026

Copy link
Copy Markdown
Collaborator Author

Addressed in da06be2.

Added a structured SimpleAggregateFunction arm to fuzzDataType. It is detected via type->getCustomName() (it is a custom-named type, not a plain IDataType), placed before the structural Array/Tuple/Nullable/... arms so those do not match its storage type and strip the custom name. The arm recursively fuzzes each argument type, keeps the aggregate name and parameters, re-validates through AggregateFunctionFactory + checkSupportedFunctions, and rebuilds via createSimpleAggregateFunctionType, so a schema like SimpleAggregateFunction(sumMap, Tuple(Array(String), Array(UInt64))) now fuzzes its nested argument types again while staying structurally valid and round-trippable (no generic-expression injection). If the aggregate rejects a fuzzed argument set it falls back to the original type.

Gate f/g audit of the remaining parametric families: plain AggregateFunction already has a structured arm (recurses into arg types + swaps the aggregate name); Object/JSON, QBit, Variant, Tuple (named + unnamed), Map, Nullable, LowCardinality, Array, Enum, Decimal, FixedString, DateTime64, Time64, Dynamic all have arms. Nested is Array(Tuple(...)) sugar (covered by the Array/Tuple arms). No remaining parametric family loses child-fuzzing coverage.

Test: extended the deterministic gtest QueryFuzzer.MalformedDataTypeArgumentRoundTrip with SimpleAggregateFunction(sum, UInt64) and SimpleAggregateFunction(sumMap, Tuple(Array(String), Array(UInt64))) round-trip forms, and added a SimpleAggregateFunction column to the 04344 smoke fixture.

Verified locally (debug, Build ID B299296CE28D5B8B1330553B382A2C22974AD388): clickhouse + unit_tests_dbms built clean, the gtest passes, and 500 --query-fuzzer-runs over the JSON + QBit + SimpleAggregateFunction CREATE kept the server alive with zero LOGICAL_ERROR / Inconsistent-AST / crash markers.

Pre-PR validation gate (click to expand)
# Item Result
a Deterministic repro Yes. gtest builds Nullable(multiply(2,3)) (ASTDataType with ASTFunction arg = the pre-fix shape) and asserts it is NOT round-trippable via ParserDataType; RNG-independent.
b Root cause explained Making ASTDataType opaque to the generic walk removed all mutation for any parametric family lacking a structured fuzzDataType arm. SimpleAggregateFunction is a custom-named type with no arm, so its name/params/nested arg types stopped being fuzzed.
c Fix matches root cause Adds the missing structured arm; no other mechanism touched.
d Test intent preserved / new tests 04344 smoke retained + extended; deterministic gtest extended with SimpleAggregateFunction forms.
e Demonstrated both directions gtest: malformed shape fails round-trip, valid SimpleAggregateFunction forms pass. Fuzz smoke: server-alive over the extended fixture.
f Fix general, not narrow Audited every parametric family; only SimpleAggregateFunction was uncovered. Plain AggregateFunction/Object/QBit/Variant/Tuple/Map/etc. already covered. Nested = Array(Tuple) sugar.
g Generalizes across inputs Arm handles any supported aggregate + any nested arg types; recurses via fuzzDataType and re-validates, falling back to the original type when the aggregate rejects the fuzzed args.
h Backward compatible N/A (fuzzer-only change, no user-visible behavior, format, or setting change).
i Invariants preserved Arm runs before the structural arms so the custom name is not stripped; result re-validated through the factory so only structurally valid, round-trippable SimpleAggregateFunction types are produced.

Session id: cron:clickhouse-worker-slot-4:20260708-122900

Comment thread src/Common/QueryFuzzer.cpp Outdated
Comment thread src/Common/QueryFuzzer.cpp Outdated
/// QBit only accepts Int8/BFloat16/Float32/Float64 element types and a (dimension, stride) pair
/// where dimension % stride == 0 and, when actually strided, stride % 8 == 0. We mutate within
/// those bounds rather than handing the argument list to the generic fuzzer.
static const DataTypePtr qbit_element_types[]

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this code could be shared with random type generation one?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@groeneai can you check this?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 53a8aa7. The genuinely duplicated part was the QBit construction (element-type set + dimension/stride constraints), which was near-identical in both functions. Extracted it into a makeRandomQBit() helper called by both fuzzDataType and getRandomType, so the stride rules live in one place (-45/+31 net).

I kept the other arms separate on purpose, since they do different things:

  • Object/JSON: getRandomType builds a bare DataTypeObject(JSON) with no params; the fuzz arm mutates the typed-path element types + max_dynamic_paths/types of an existing node while preserving its SKIP paths. There is no shared construction to factor.
  • (Simple)AggregateFunction: getRandomType synthesizes from scratch (with a count fallback on rejection); the fuzz arms mutate an existing aggregate, preserving its name/parameters and only re-validating fuzzed argument types. Folding these into one helper would obscure the generate-vs-mutate distinction rather than reduce real duplication.

Build + gtest + the fuzz smoke (JSON/QBit/SimpleAggregateFunction CREATE, 500 runs, server stays alive) all pass.

…mType

Extract the duplicated QBit-building logic (element type + dimension/stride
constraints) into a single makeRandomQBit helper used by both fuzzDataType
and getRandomType, so the stride constraints live in one place.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Comment thread src/Common/QueryFuzzer.cpp Outdated
/// Fuzz the typed-path element types (recursively) and the numeric parameters, keeping the
/// SKIP paths / SKIP REGEXP lists intact so the result stays a structurally valid, round-trippable
/// DataTypeObject (its getName() re-parses via ParserDataType to the same type).
std::unordered_map<String, DataTypePtr> typed_paths = type_object->getTypedPaths();

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@groeneai this code should also be shared

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in f6966fc. Extracted makeRandomObject() (builds a JSON Object with randomized max_dynamic_paths/max_dynamic_types within the parser limits); both getRandomType (empty paths) and the fuzzDataType Object arm (fuzzed typed paths + preserved SKIP lists) call it now.

Comment thread src/Common/QueryFuzzer.cpp Outdated
if (type->hasCustomName())
{
if (const auto * type_simple_aggr
= typeid_cast<const DataTypeCustomSimpleAggregateFunction *>(type->getCustomName());

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@groeneai this code should also be shared

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in f6966fc. Extracted makeAggregateFunctionType(name, args, params, simple) which runs the factory get + build (and, for the simple form, checkSupportedFunctions) and returns nullptr if the aggregate rejects the args. Shared by three sites now: getRandomType AggregateFunction case, and the fuzzDataType AggregateFunction and SimpleAggregateFunction arms.

@PedroTadim
PedroTadim disabled auto-merge July 29, 2026 13:44
Requested in review: keep the data-type machinery separate from the rest of the
fuzzer so it can be reviewed on its own.

Pure code motion. fuzzDataType, getRandomType and their nine helpers move from
QueryFuzzer.cpp to the new QueryFuzzerDataTypes.cpp, together with the two
function-local tables and the timezone list they use. QueryFuzzer.cpp keeps the
guard in fuzz() that this PR is actually about, so the fix and the type
machinery are no longer read as one block.

src/Common is globbed by add_headers_and_sources, so the new translation unit
needs no CMake change.

The only edit that is not a move: swapAggrs is pre-existing code in
QueryFuzzer.cpp used by both units now, so it is reachable through a
swapAggregateNames accessor, in the same style as the geoAliasNames accessor
this PR already added. Its definition and the fuzz() use of it are untouched.
Three includes this PR had added to QueryFuzzer.cpp moved with their code and
are dropped from it.

Verified behaviour preserving rather than assumed: cityHash64(groupArray(query))
over seeded fuzzQuery output for three fixtures covering the argument bearing
types is identical before and after the move. Perturbing makeRandomDateTime
inside the moved region changes one of the three hashes, so the comparison is
discriminating and not vacuous.
@groeneai

Copy link
Copy Markdown
Collaborator Author

CI finish ledger - f46aa21

Every failure below has an owner: a fixing PR (ours or external), or a full-effort fix task
whose fixing-PR link will be posted here when it opens. Only CH Inc sync is exempt.

Check / test Reason Owner / fixing PR
Integration tests (amd_tsan, 2/6) / test_s3_plain_rewritable test_projections[s3_plain_rewritable_with_metadata_cache] ThreadSanitizer data race on a shared_ptr<FsNode> control word under disjoint mutex sets, in the use_count() copy-on-write predicate at FsSnapshot.cpp:39 reached from clonePath a fix task is created (investigating at full effort, fixing-PR link to follow here)
Integration tests (amd_tsan, 2/6) / test_s3_plain_rewritable test[cache_s3_plain_rewritable-data/] Code: 210 NETWORK_ERROR Connection refused, victim of the sanitizer-aborted server above, same event same owner as the row above
Integration tests (amd_tsan, 2/6) / test_s3_plain_rewritable test[s3_plain_rewritable_with_metadata_cache-data_with_cache/] Code: 210 NETWORK_ERROR Connection refused, victim of the same event same owner as the row above

All 174 check-runs on this commit are completed; this is the only failing check.

Not caused by this PR. This PR touches src/Common/QueryFuzzer.{cpp,h},
src/Common/QueryFuzzerDataTypes.cpp, src/DataTypes/DataTypeAggregateFunction.h and its own
04344_ast_fuzzer_datatype_arg test pair, nothing in the Disks or plain-rewritable metadata
subsystem. The same race, with a byte-identical stack, also fired the same day on unrelated
PR #96199 (commit cd143e56627fbed82d9a5295e4733fe6c060efa5) and on #110916 and #112329.

Onset is sharp: a 60-day CIDB sweep keyed on the clonePath frame returns hits on exactly one day,
2026-07-29, across those four unrelated PRs, and zero hits in the 59 days before. My working root
cause is the reference-count copy-on-write introduced by #111883, which replaced an unconditional
FsNode clone with if (node.use_count() > 1); use_count is not an exclusivity oracle across
threads, so two transactions can alias and then mutate the same node under different mutexes. I am
investigating that in the fix task and will open a separate PR rather than touching #111883.

Session id: cron:our-pr-ci-monitor:20260729-223000

Comment thread src/Common/QueryFuzzer.cpp
@groeneai

Copy link
Copy Markdown
Collaborator Author

Fixing PR for the plain_rewritable FsNode data race: #112557

That is a revert of #111883 by @ alexey-milovidov, merged 2026-07-30T09:07:30Z, which restores the
unconditional std::make_shared<FsNode>(*node) in clonePath and deletes the
use_count() > 1 copy-on-write predicate. He also opened a targeted fix, #112559, with the same
root cause; @ Michicosun closed it as fixed by #112557.

The race is verified gone on trunk, not just merged: keyed on the FsSnapshot.cpp frame plus
data race, CIDB returns 0 rows on any commit containing the revert, while in the same window the
tsan jobs ran 10160 passing plain_rewritable test rows across 255 distinct commits with 0
failures, so the silence is real and not a test that stopped running. All 7 carrier commits from
the 07-29/07-30 cluster contain the reverted helper; the last hit is 6d6b2615aff289 at
07-30 06:22 UTC, which is not a descendant of the revert.

This head predates the revert, so the red here clears once the branch picks up master.

One correction to my earlier ledger note on the onset: I flagged two possible pre-#111883 hits
(#64184 on 07-22, #95071 on 07-28). Neither is this bug. Both are Build (arm_tidy) rows whose
context merely contains the clang-tidy-cache log line for the object file FsSnapshot.cpp.o, with
no stack frame and no sanitizer report. The onset is exactly 2026-07-29, as the clonePath-keyed
sweep originally indicated.

…09706

The only conflict was src/Common/QueryFuzzer.cpp. This branch moved the
data-type fuzzing (fuzzDataType, getRandomType and their helpers) into the
new src/Common/QueryFuzzerDataTypes.cpp, so master's change to
getRandomType's Map case presented as a delete/modify conflict rather than
overlapping text.

Master's new guard (03fdf14, "Query fuzzer: do not compose a Map with
an invalid key type") is carried over verbatim into its new home in
QueryFuzzerDataTypes.cpp, so the Nullable/LowCardinality(Nullable) Map key
rejection is preserved. The pre-existing fuzzDataType Map guard is a distinct
site and was already present in the merge base.

The other seven master hunks land in QueryFuzzer.cpp unchanged: the DateLUT
include, the ASTStreamSettings cursor field rename with its buildCursorTree
calls, and fake_time_for_view switching to a formatted timestamp.
The branch had to enumerate every ASTDataType subclass by hand because
typeid_cast matches the exact type only, so a subclass missing from the
list would fall through to the generic expression fuzzer and reintroduce
the invalid argument lists of ClickHouse#109706. dynamic_cast covers the whole
hierarchy, so the condition is one cast instead of three and a future
subclass cannot silently escape it.

Behaviour is unchanged: ASTDataType, ASTTupleDataType and ASTEnumDataType
are the only subclasses today.
@groeneai

groeneai commented Aug 1, 2026

Copy link
Copy Markdown
Collaborator Author

CI finish ledger — d0c97df

CI is fully finished on this head (174 check-runs, 0 incomplete, Finish Workflow success) with no failures. The Integration tests (amd_tsan) FsSnapshot data race that owned the previous head's ledger line does not recur here.

Nothing to attribute, so there is no owner column to fill.

Session id: cron:our-pr-ci-monitor:20260801-180000

@clickhouse-gh

clickhouse-gh Bot commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

LLVM Coverage Report

Metric Baseline Current Δ
Lines 86.50% 86.50% +0.00%
Functions 91.90% 91.90% +0.00%
Branches 78.70% 78.80% +0.10%

Changed lines: Changed C/C++ lines covered: 340/355 (95.77%) · Uncovered code

Full report · Diff report

@groeneai

groeneai commented Aug 2, 2026

Copy link
Copy Markdown
Collaborator Author

CI finish ledger for 347be61

Every failure below has an owner: a fixing PR, or a full-effort fix task whose fixing-PR link will be posted here when it opens.

Check / test Reason Owner / fixing PR
Stateless tests (amd_tsan, s3 storage, parallel, 2/3) / Server died + logical error Bad cast from type DB::ColumnNullable to DB::ColumnString (STID 5793-6473) pre-existing trunk bug: SerializationString::serializeBinaryBulkWithSizeStream typeid_casts unconditionally to ColumnString while the part writer recorded a Nullable column #112501 (external, open)

174 of 174 check-runs completed, Finish Workflow green. The two rows above are ONE event, not two: the Server died row is the collateral of the same abort, and the suite itself reports Failed: 0, Passed: 668.

Not caused by this diff, which stops the AST fuzzer injecting expressions into data-type arguments. Breadth over 30 days on the message is 48 rows across 14 checks with master rows on 8 of them, so it is reachable on trunk independently of this branch.

Session id: cron:our-pr-ci-monitor:20260802-010000

# Conflicts:
#	src/Common/QueryFuzzer.cpp
Comment thread src/Common/QueryFuzzerDataTypes.cpp
Making ASTDataType opaque to the generic expression fuzzer removed the only
mutation path that used to reach the SKIP clauses of a JSON type: the skipped
path is a compound identifier and the skipped regexp a string literal, both of
which the generic walk mutated on its way down (the ASTLiteral arm rewrites
literal->value via fuzzField). The DataTypeObject arm that took ownership
re-fuzzed the typed-path types and the numeric parameters but carried both SKIP
collections through verbatim, so JSON(SKIP a, SKIP REGEXP 're') stopped being
mutated at all.

Add a structured mutation for each collection. Replacements are drawn from a
fixed table so they stay inside the grammar the type requires: identifier-shaped
values for a skipped path, RE2-compilable ones for a skipped regexp (the
DataTypeObject constructor throws CANNOT_COMPILE_REGEXP otherwise). This keeps
the result round-trippable, which is the property this PR exists to preserve --
mutating into something ParserDataType cannot re-parse is what produced the
inconsistent-AST-formatting abort in the first place.

The paths_to_skip set is iterated in unspecified order, so it is sorted before
an element is picked to keep a run reproducible for a given seed. A mutated
skipped path can collide with a typed path prefix, which the constructor
rejects; that throw is already absorbed by the enclosing try/catch, which falls
back to the unmutated type.

Measured over 12 seeds of the fuzzQuery table function on
JSON(a.b UInt32, SKIP zzz, SKIP REGEXP '^qqq.*$'): 80 JSON rows, both clauses
mutated (8 distinct regexps, 6 distinct paths) while the originals still occur,
confirming the mutation fires without displacing the input shape.
@groeneai

groeneai commented Aug 3, 2026

Copy link
Copy Markdown
Collaborator Author

CI finish ledger - 0ae0f0f

Every failure below has an owner: a fixing PR (mine or external), or a full-effort fix task
whose fixing-PR link will be posted here when it opens. Only CH Inc sync is exempt.

Check / test Reason Owner / fixing PR
Stress test (amd_tsan) / Received signal 6 (internal) (STID 2009-2ba7) spurious std::abort from the AST fuzzer's own loop detector on a recycled node address #113114 (external, open)
Build (arm_release) / Post Hooks telemetry INSERT in build_profile_hook.py, job-level check_status = success a fix task is moved to pending (investigating at full effort, fixing-PR link to follow on this PR)

175/175 check-runs completed; Finish Workflow completed successfully.

On the first row: the stack sits in QueryFuzzer::fuzz at the debug_visited_nodes duplicate-address
check, which calls std::abort when it sees one address twice. #113114 shows the check is unsound
because fuzzing legitimately frees nodes it has already visited, after which the allocator can hand the
same address to a later node, so it fires with no loop present. That is a pre-existing defect in the
detector, not a consequence of this PR: my diff changes which nodes the fuzzer descends into for data
types, and cannot make a freed address be reused. The signature is 5 hits across 5 distinct commits in
30 days, all of them on 2026-08-03 and 4 of them not mine, including one on master.

Session id: cron:our-pr-ci-monitor:20260803-163000

The fixture carried a SKIP path but no SKIP REGEXP, so fuzzObjectPathRegexpsToSkip
was never reached for this input and the new mutation had no coverage here.

Note this makes the clause reachable; it does not make the test a discriminating
oracle for it. An unsafe replacement cannot be caught by this test by
construction: DataTypeObject validates the regexp in its constructor, and the
enclosing catch in the DataTypeObject arm falls back to the unmutated type, so a
bad candidate degrades coverage rather than emitting an unparseable type.
Verified by injecting an RE2-invalid candidate and confirming it never reaches
the output (0 of 83 generated JSON types) while the test stays green.
@groeneai

groeneai commented Aug 4, 2026

Copy link
Copy Markdown
Collaborator Author

CI finish ledger - 4aa045a

CI is fully finished on this head: 175 check-runs, 0 incomplete, Config Workflow and
Finish Workflow both success, past the 20-minute ingestion buffer. One failure, and it has an owner.

Check / test Reason Owner / fixing PR
Stress test (arm_tsan) / Received signal 6 (internal) (STID 2009-2ba7) trunk regression in the fuzzer's own loop-detection guard #113114 (external, open)

Not caused by this PR. The abort is QueryFuzzer::fuzz's loop check ("The AST node ... was already visited before" -> std::abort) reached through fuzzCreateQuery, and the same stack fires on
master (dd1e53dfe2, Stress test (arm_release), 11:59Z) and on 6 unrelated PRs across
arm_release, amd_tsan, amd_msan and arm_debug -- 10 hits in one day, and zero in the
preceding 13 days. #113114 identifies the cause: the guard stores bare node addresses, so an
address freed when fuzzing legitimately drops part of the query can be recycled by a later node and
read as a loop. Its diff holds each visited node alive
(unordered_set<const IAST *> -> unordered_map<const IAST *, ASTPtr>), which makes pointer
identity valid for the whole fuzzMain call.

The corpus query confirms the match: my failing run carries
log_comment = '03305_parallel_with.sql-test_q5m6x5b8d5n1', the same 03305_parallel_with.sql
CREATE TABLE ... PARALLEL WITH CREATE TABLE ... shape #113114 analyses.

#113114 is open and is not an ancestor of this head (QueryFuzzer.h:218 here still declares
std::unordered_set<const IAST *> debug_visited_nodes), so this build predates the fix rather than
failing with it applied.

Session id: cron:our-pr-ci-monitor:20260804-003000

Comment thread tests/queries/0_stateless/04344_ast_fuzzer_datatype_arg.sql Outdated
Requested by the reviewer: the test is not needed. The fuzzer fix in
src/Common/QueryFuzzer.cpp and src/Common/QueryFuzzerDataTypes.cpp is
unchanged; only the stateless test and its reference file are dropped.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Comment thread src/Common/QueryFuzzerDataTypes.cpp
makeRandomObject is used both to synthesize a fresh JSON type and to rebuild
a mutated one. In the mutate path the 25% randomization of max_dynamic_paths /
max_dynamic_types fell back to the compile-time defaults rather than to the
source type's own values, so a source such as
JSON(max_dynamic_paths=0, max_dynamic_types=1, p UInt32) lost that regime on
most visits even when only a typed path or a SKIP clause was being mutated.
Since ASTDataType is opaque to the generic expression fuzzer, this helper is
the only remaining mutation path for those parameters, so the regime was
simply unreachable.

Thread the source limits through as optional parameters: an unfired
randomization now keeps them, a fired one still explores the whole range, and
the generate path passes nothing and keeps using the defaults. This mirrors
how makeAggregateFunctionType already threads the parsed serialization
version.

Measured over 12 seeds against
JSON(max_dynamic_paths=0, max_dynamic_types=1, p UInt32), counting emitted JSON
types (doGetName only prints a limit clause when it differs from the default,
so a reset limit is visible as the clause disappearing):

  before: 86 types, max_dynamic_paths reset 59, max_dynamic_types reset 58
  after:  50 types, max_dynamic_paths reset  0, max_dynamic_types reset  1

The single remaining reset is the randomization legitimately drawing the
default value. All 23 distinct emitted JSON types reparse to themselves.
@groeneai

groeneai commented Aug 4, 2026

Copy link
Copy Markdown
Collaborator Author

CI finish ledger - f2e9025

Every failure below has an owner: a fixing PR (mine or external), or a full-effort fix task
whose fixing-PR link will be posted here when it opens. Only CH Inc sync is exempt.

Check / test Reason Owner / fixing PR
Stress test (amd_msan) / Hung check failed, possible deadlock found DatabaseFilesystem::mutex convoy: 58 of the 61 stuck queries reference DatabaseFilesystem, the lead is a 02722_database_filesystem SELECT count(...) at 1894s and the holders are DROP DATABASE ... SYNC #112495 (mine, open)
AST fuzzer (amd_debug, targeted, old_compatibility) / Logical error: Reading from materialized CTE 'A' before its materialization completed - DelayedPortsProcessor gate is missing in the query plan (STID 2467-2c2d) chronic trunk planner bug, 45 rows / 37 PRs / 3 master in 30d #102308 (external, open)
AST fuzzer (amd_tsan) / Logical error: Invalid number of columns in chunk pushed to OutputPort. Expected 2, found 5 (STID 2270-2ea4) trunk logical error, #96656 family (one abort site, several inlining vintages) #111401 (mine, open)
Stress test (arm_msan) / Logical error: 'column->size() == num_rows' (STID 2508-562d) trunk logical error in MergeTreeRangeReader::ReadResult::checkInternalConsistency on a shared Nested offsets read #113225 (mine, open)
Build (arm_release) / Post Hooks build-profile telemetry INSERT INTO build_time_trace gets HTTP 500 from LogCluster; check_status is success and the check is non-gating a fix task is moved to pending (investigating at full effort - fixing PR link to follow here)

Not caused by this PR. The diff only changes how the AST fuzzer mutates ASTDataType argument lists
(src/Common/QueryFuzzer.{cpp,h}, src/Common/QueryFuzzerDataTypes.cpp,
src/DataTypes/DataTypeAggregateFunction.h). None of the four aborts involves an ASTDataType
argument list: two are planner/pipeline logical errors on ordinary fuzzed queries, one is a
MergeTree reader consistency assert, and the hung check is a database-layer lock convoy. No frame of
the diff appears in any of them.

Session id: cron:our-pr-ci-monitor:20260804-173000

@alexey-milovidov
alexey-milovidov added this pull request to the merge queue Aug 5, 2026
@groeneai

groeneai commented Aug 5, 2026

Copy link
Copy Markdown
Collaborator Author

CI finish ledger - 58f118e

Every failure below has an owner: a fixing PR (mine or external), or a full-effort fix task whose
fixing-PR link will be posted here when it opens. Only CH Inc sync is exempt.

Check / test Reason Owner / fixing PR
Build (arm_release) / Post Hooks infra: build_profile_hook.py telemetry INSERT rejected with LogCluster query failed with code 500 after the build itself succeeded #113409 (external, open)

All 174 check-runs completed and Finish Workflow succeeded; the status rollup on this commit is
green. Post Hooks does not gate the check it runs under, so it is invisible to the check-runs API
and shows up only as a per-test row.

Session id: cron:our-pr-ci-monitor:20260805-230000

Merged via the queue into ClickHouse:master with commit 2fb5262 Aug 5, 2026
179 checks passed
@robot-ch-test-poll4 robot-ch-test-poll4 added the pr-synced-to-cloud The PR is synced to the cloud repo label Aug 6, 2026
alexey-milovidov added a commit to groeneai/ClickHouse that referenced this pull request Aug 11, 2026
Master commit 7cc0440 (AST-fuzzer follow-up of ClickHouse#109713) added its own
`getVersionIfExplicit` to `DataTypeAggregateFunction`; this branch already
provides an identical definition. Keep a single one.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

can be tested Allows running workflows for external contributors pr-ci pr-synced-to-cloud The PR is synced to the cloud repo

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[CI crash] TCPHandler runImpl caused unexpected termination

4 participants