[fix](docker)(case) Fix kerberized-HDFS p2 suites: SASL client property, Paimon fixture, iceberg root-cause reporting - #66433
Merged
morningman merged 3 commits intoAug 5, 2026
Conversation
apache#65564 rebuilt the Kerberos test environment and created `kerberos/conf/hdfs-site.xml.tpl` with dfs.data.transfer.protection=authentication The environment it replaced carried its server configuration inside the starburstdata image, so this server-side requirement is new. A client that does not set the same property sends a plain writeBlock op, `SaslDataTransferServer` does not find the SASL magic number and closes the socket, the client excludes the only DataNode, and the next `addBlock` has no target left. Metadata operations only reach the NameNode and the HMS, so `CREATE DATABASE`/`CREATE TABLE` still succeed and only the write fails: could only be written to 0 of the 1 minReplication nodes. There are 1 datanode(s) running and 1 node(s) are excluded in this operation. with `jni.log` showing why the DataNode was excluded - the socket was established and the server closed it: java.net.SocketException: Connection reset at DataStreamer.createBlockOutputStream(DataStreamer.java:1931) Excluding datanode DatanodeInfoWithStorage[<ip>:9566,...] (Through iceberg the same failure surfaces as `IllegalArgumentException: Self-suppression not permitted`, which is what `TableMetadataParser.internalWrite` turns it into when try-with-resources calls `Throwable.addSuppressed(this)`.) apache#65564 set the client property in the four `external_table_p0/kerberos` suites but not in the p2 suites that talk to the same kerberized HDFS. The gap was diagnosed on the branch-4.0 external pipeline, where the backport (apache#65939) turned `hive_on_hms_and_dlf` and `iceberg_on_hms_and_filesystem_and_dlf` red on every run with exactly these two signatures; master carries the identical omission. Fix: set `dfs.data.transfer.protection` in every property block that points at the kerberized HDFS. A scan of `:8520`/`:8620` across the whole suite tree finds seven such blocks still missing it: two each in `hive_on_hms_and_dlf`, `iceberg_on_hms_and_filesystem_and_dlf` and `test_paimon_hms_catalog`, plus one in `hdfs_all_test`, which writes to HDFS but currently returns early because `refactor_params_hdfs_kerberos_test` is unset - it would fail the moment it is enabled. The kerberos block of `test_information_schema_timezone` has the same gap but is disabled by an unrelated TODO, so it is left alone. `test_paimon_hms_catalog` needs one more thing to go green: the lightweight kerberos container starts an empty metastore, so `hdfs_db` does not exist there. That is an environment change and comes in a separate commit. Signed-off-by: morningman <yunyou@selectdb.com>
Creating an Iceberg table on a kerberized HDFS whose DataNode rejected the
write surfaces only
Self-suppression not permitted
to the user, which says nothing about what went wrong (diagnosed on
branch-4.0; master wraps the same way through `DorisConnectorException`).
`TableMetadataParser.internalWrite` writes the metadata JSON inside
try-with-resources, and `DFSOutputStream` throws the same exception instance
from `write` and from `close` once its pipeline has failed, so the generated
`addSuppressed` call becomes `addSuppressed(this)` and `Throwable` replaces
the failure with `IllegalArgumentException: Self-suppression not permitted`.
The real error is one level down:
Caused by: RemoteException: ... could only be written to 0 of the 1
minReplication nodes. There are 1 datanode(s) running and 1 node(s) are
excluded in this operation.
`IcebergConnectorMetadata` passes the exception as the cause, so the
frontend log keeps that chain, but the message returned over the MySQL
protocol is built from `e.getMessage()` and therefore carries only the
wrapper.
Every path that commits Iceberg table metadata goes through the same parser
and can be masked the same way. This switches all nineteen commit paths -
`createTable`, the column and schema updates (including the nested-column
and comment variants), the branch and tag operations, and the
partition-field operations - to `ExceptionUtils.getRootCauseMessage(e)`.
The legacy `IcebergMetadataOps` already reported the root cause for the
partition-field trio; the connector rewrite lost that, so this also
restores it.
The surrounding message is a literal prefix that already names the
operation and the object, so replacing only the trailing part loses no
context, and no regression case asserts on this text.
Read paths such as `getTableHandle`, `loadTable` and `listTableNames`
cannot hit self-suppression and keep `e.getMessage()`. So do `dropTable`,
`renameTable`, `dropView` and the database operations: they delete or
alter existing objects instead of writing a metadata file through the
parser.
Signed-off-by: morningman <yunyou@selectdb.com>
…tore The hdfs_kerberos and hdfs_new_kerberos cases of test_paimon_hms_catalog fail with `Unknown database 'hdfs_db'`, and the suite aborts on the first of them. The lightweight Kerberos environment from apache#65564 starts an empty metastore - `schematool -dbType derby -initSchema` and nothing else. The environment it replaced provisioned the Paimon fixture on every start: hadoop fs -put /tmp/paimon_data/* /user/hive/warehouse/ hive -f /usr/local/sql/create_paimon_hive_table.hql along with the Paimon storage-handler jar it downloaded into the image's hive auxlib. None of that survived. `paimon_data/hdfs_db.db/` is still in the tree but the compose file never mounted it, so it has been dead weight since the migration. (Diagnosed on the branch-4.0 external pipeline, whose backport of the same migration broke the same cases; the branch-4.0 counterpart of this fix is in apache#66417.) This restores the fixture on kerberos1 only - it owns metastore 9583, which is the only one the suite talks to, so kerberos2 stays as light as it is today. Both containers run the same entrypoint with the same rendered env switch, so the fixture is additionally gated on the container role (HOST): kerberos2 has neither the mounts nor a consumer, and without that gate `set -e` would abort it the moment the switch is on. Gating on the role rather than on the mounts keeps a broken mount on kerberos1 failing loudly. `hdfs_db` is backed by `paimon_data`, uploaded into HDFS before the metastore is advertised. The table declares no columns, so the metastore derives the schema through the storage handler (`metastore.storage.schema.reader.impl` is `SerDeStorageSchemaReader`); `HIVE_AUX_JARS_PATH` is therefore exported before the metastore starts, not just around the DDL. Unlike the branch-4.0 sibling, this does not register `ali_db` over OSS - and the environment this restores never had it on master: the pre-apache#65564 HQL provisioned exactly `hdfs_db`. The oss_hms_kerberos case that reads `ali_db` through this metastore exists only on branch-4.0, master's ali_db cases go through the Hive3 metastore, and the golden output here carries only the hdfs_kerberos and hdfs_new_kerberos blocks. Skipping it keeps the jindo/aliyun jars and the OSS credentials out of the kerberos stack and removes a start-time network dependency no case exercises. The fixture runs before `DORIS_KERBEROS_READY`, because `run-thirdparties-docker.sh` releases the pipeline on that marker and anything published afterwards would race the suites. A failure aborts the container instead of handing out an environment silently missing `hdfs_db`. `enablePaimonHms` is read from `hive-3x_settings.env`, the file the pipeline already patches, rather than copied into `kerberos*_settings.env` where the two would drift and one stack would quietly lose the fixture. It is sourced in a subshell so that the Hive3 ports cannot leak into the Kerberos settings sourced immediately after. Verified locally: `bash -n` passes on both scripts; the templates and the compose file render with no leftover placeholders and the rendered XML parses; the rendered compose gives kerberos1 exactly the three extra mounts and leaves kerberos2 untouched; the rendered env files carry the right HOST and enablePaimonHms per container; the settings resolution was exercised for the enabled, disabled and missing-file cases with an assertion that the Hive3 ports do not leak; and the entrypoint gate was exercised for both container roles and both switch values. The golden output already contains the hdfs_kerberos and hdfs_new_kerberos blocks, so no baseline has to be regenerated. The container itself needs the CI environment and has not been run. Signed-off-by: morningman <yunyou@selectdb.com>
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
morningman
added a commit
to morningman/doris
that referenced
this pull request
Aug 4, 2026
Both kerberos containers run entrypoint-hive-master.sh, and hadoop-hive-1.env
and hadoop-hive-2.env are rendered from the same template with the same host
environment, so kerberos2 receives enablePaimonHms=true whenever kerberos1
does. The fixture's mounts (sql/, paimon_data/, auxlib/) exist on kerberos1
only, so under `set -euo pipefail` kerberos2 would abort at
hdfs dfs -put -f /opt/doris/paimon_data/* /user/hive/warehouse/
the moment the pipeline turns the switch on, taking down every case that uses
the second realm.
Gate the fixture on the container role (HOST=hadoop-master, set per container
by kerberos*_settings.env) in addition to the switch. Gating on the role
rather than on the mounts keeps a broken mount on kerberos1 failing loudly,
while kerberos2 skips deterministically.
Found while porting this fix to master; the master PR (apache#66433) carries the
same gate.
Signed-off-by: morningman <yunyou@selectdb.com>
Contributor
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 28859 ms |
Contributor
TPC-DS: Total hot run time: 166574 ms |
Contributor
ClickBench: Total hot run time: 23.93 s |
Contributor
FE Regression Coverage ReportIncrement line coverage |
suxiaogang223
approved these changes
Aug 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: close #xxx
Related PR: #65564, #66313, #66417
Problem Summary:
Master port of the remaining external-regression gaps diagnosed on the
branch-4.0 pipeline (#66417). #66313 already carried the keytab and baseline
fixes to master; this PR carries the rest. All three defects were verified to
exist on master by inspection — master shares the same code and the same
lightweight Kerberos environment (#65564, which #65939 backported to 4.0).
1. Kerberos HDFS writes fail in the p2 suites (SASL client property missing)
#65564 created
kerberos/conf/hdfs-site.xml.tplwithdfs.data.transfer.protection=authentication. A client that does not set thesame property sends a plain
writeBlockop,SaslDataTransferServercloses thesocket, the client excludes the only DataNode, and the write fails with
(through iceberg this surfaces as
IllegalArgumentException: Self-suppression not permitted). Metadata operations only reach the NameNode and the HMS, soCREATE DATABASE/CREATE TABLEstill succeed and only the write fails.#65564 set the client property in the four
external_table_p0/kerberossuitesbut not in the p2 suites that talk to the same kerberized HDFS. On the
branch-4.0 pipeline this turned
hive_on_hms_and_dlfandiceberg_on_hms_and_filesystem_and_dlfred on every run; master carries theidentical omission.
Fix: set
dfs.data.transfer.protectionin every property block that points atthe kerberized HDFS. A scan of
:8520/:8620across the whole suite tree findsseven such blocks still missing it: two each in
hive_on_hms_and_dlf,iceberg_on_hms_and_filesystem_and_dlfandtest_paimon_hms_catalog, plus onein
hdfs_all_test(currently returns early becauserefactor_params_hdfs_kerberos_testis unset — it would fail the moment it isenabled). The kerberos block of
test_information_schema_timezonehas the samegap but is disabled by an unrelated TODO, so it is left alone.
2.
test_paimon_hms_catalog:Unknown database 'hdfs_db'The lightweight Kerberos environment starts an empty metastore —
schematool -initSchemaand nothing else. The environment it replaced uploadedpaimon_datainto HDFS and registeredhdfs_dbthroughcreate_paimon_hive_table.hqlon every start;paimon_data/hdfs_db.db/isstill in the tree but the compose file never mounted it. This restores the
fixture on
kerberos1only (it owns metastore 9583, the only one the suitetalks to), runs it before
DORIS_KERBEROS_READYso nothing races the suites,and aborts the container on failure instead of handing out an environment
silently missing
hdfs_db.Two deliberate differences from the branch-4.0 sibling commit:
hdfs_dbonly, noali_db/OSS. Master's pre-[improvement](regression) Reduce Kerberos test environment memory #65564 HQL provisionedexactly
hdfs_db; theoss_hms_kerberoscase that readsali_dbthroughthe kerberized metastore exists only on branch-4.0, master's
ali_dbcasesgo through the Hive3 metastore, and the golden output here carries only the
hdfs_kerberos/hdfs_new_kerberosblocks. This keeps the jindo/aliyunjars and OSS credentials out of the kerberos stack.
HOST). Both kerberoscontainers run the same entrypoint with the same rendered env switch, but
only
kerberos1carries thesql//paimon_data//auxlib/mounts — withoutthe gate,
set -ewould abortkerberos2the momentenablePaimonHmsison. (The branch-4.0 sibling needs the same gate; a follow-up on branch-4.0: [fix](docker)(case) Fix the remaining 4 external regression failures: kerberos HDFS SASL, Paimon fixture, file cache flakiness #66417 will
add it.)
3.
IcebergConnectorMetadata: report the root cause of failed metadata commitsTableMetadataParser.internalWritewrites the metadata JSON insidetry-with-resources, and
DFSOutputStreamthrows the same exception instancefrom
writeand fromcloseonce its pipeline has failed, soThrowable.addSuppressed(this)replaces the failure withIllegalArgumentException: Self-suppression not permitted. The FE log keeps thechain, but the message returned over the MySQL protocol is built from
e.getMessage()and carries only the wrapper.This switches all nineteen metadata-commit paths —
createTable, thecolumn/schema updates (including the nested-column and comment variants), the
branch/tag operations, and the partition-field operations — to
ExceptionUtils.getRootCauseMessage(e). The legacyIcebergMetadataOpsalready reported the root cause for the partition-field trio; the connector
rewrite lost that, so this also restores it. Read paths and the drop/rename
operations cannot hit self-suppression and keep
e.getMessage().Intentionally not ported from #66417: the file-cache de-flake. Master's
test_file_cache_features(rewritten in #66313's lineage) already waits fordisk_resource_limit_mode/need_evict_cache_in_advanceto clear beforereturning, which closes the residual-state window at its source.
Verification (local):
fe-connector-icebergcompiles and passes checkstyle(
mvn -pl fe-connector/fe-connector-iceberg package); the four changed suitescompile with the offline groovy compiler;
bash -npasses on both shellscripts; the compose/env/conf templates render with no leftover placeholders,
the rendered XML parses, kerberos1 gets exactly the three new mounts while
kerberos2 is unchanged; the settings resolution and the entrypoint role-gate
were exercised for all switch/role combinations. The containers themselves need
the CI environment and have not been run.
Release note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?