HIVE-29777: Iceberg: Add GCS, ADLS, and OSS Hadoop mapping for REST vended credentials

[difin]

What changes were proposed in this pull request?

This PR extends Iceberg REST vended credential propagation beyond S3 to GCS, ADLS, and OSS. It refactors IcebergVendedCredentialUtil into provider-specific Hadoop mappers that translate Iceberg StorageCredential config keys into the correct fs.gs.*, fs.azure.*, and fs.oss.* properties. New GCS/ADLS/OSS secret keys are added to hive.conf.hidden.list so they travel via the secure Credentials channel rather than plain job properties. Unit tests cover the new mappings and round-trip behavior.

Why are the changes needed?

The original vended-credential work only propagated Hadoop mappings for S3. Iceberg REST catalogs can vend credentials for GCS, ADLS, and OSS as well, but without equivalent mapping those jobs cannot configure the corresponding Hadoop FileSystem connectors on Tez/LLAP execution paths. This change closes that gap so multi-cloud Iceberg deployments can use vended credentials consistently across providers.

Does this PR introduce any user-facing change?

Yes, for deployments using Iceberg REST catalogs with vended credentials on GCS, ADLS, or OSS. Those workloads can now propagate storage credentials to Hadoop FileSystem paths on Tez/LLAP, which was not supported before. Users on S3-only setups or without vended credentials should see no change in behavior.

How was this patch tested?

• Unit tests in TestIcebergVendedCredentialUtil were expanded to cover GCS, ADLS, and OSS Hadoop property mapping, secret routing into jobSecrets, mapper selection by URI scheme and config-key fallback, and Credentials-channel round-trip restore. Existing S3 coverage was kept and refactored to use the new mapper layout.

• New q-test driver was added for ADLS + Gravitino + Vended Credentials test: TestIcebergRESTCatalogGravitinoAdlsLlapLocalCliDriver.

• New q-test driver was added for GCP + Gravitino + Vended Credentials test: TestIcebergRESTCatalogGravitinoGcpLlapLocalCliDriver.

• The 3 drivers TestIcebergRESTCatalogGravitinoAdlsLlapLocalCliDriver, TestIcebergRESTCatalogGravitinoGcpLlapLocalCliDriver, and TestIcebergRESTCatalogGravitinoLlapLocalCliDriver run the same q-file: iceberg_rest_catalog_gravitino.q, testing vended credentials with S3, ADLS, and GCP storage providers.

• No integration coverage was added for OSS in this PR; that is planned as follow-up work.

[Copilot]

Pull request overview

Extends Hive’s Iceberg REST vended-credential propagation from S3-only to additional storage providers by introducing provider-specific Hadoop configuration mappers (GCS, ADLS, OSS), and refactoring vended-credential handling to route secrets through Hive’s Credentials channel while emitting non-secret connector properties into job properties.

Changes:

• Refactors IcebergVendedCredentialUtil to use provider-specific Hadoop mappers selected by prefix scheme (with config-key fallback).
• Adds new vended-credential parsing/mapping utilities under org.apache.iceberg.mr.hive.vended for S3/GCS/ADLS/OSS.
• Expands unit tests to cover new provider mappings and non-S3 credentials-channel round-trip behavior; updates Hive hidden config list with new Iceberg credential keys.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
iceberg/iceberg-handler/src/test/java/org/apache/iceberg/mr/hive/TestIcebergVendedCredentialUtil.java	Adds/updates unit tests for GCS/ADLS/OSS mapping and credentials-channel restore coverage.
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/vended/VendedCredentialSupport.java	Adds shared helpers for mapper selection, prefix parsing delegation, and extracting credentials from FileIO properties across providers.
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/vended/VendedCredentialPrefixUtil.java	Adds prefix/scope parsing and location→prefix normalization utilities.
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/vended/VendedCredentialHadoopMappers.java	Adds mapper selection logic by scheme then config-key namespace fallback.
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/vended/VendedCredentialHadoopMapper.java	Introduces the mapper interface for translating Iceberg credential keys to Hadoop connector keys plus required non-secret companion properties.
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/vended/S3VendedCredentialHadoopMapper.java	Implements S3 vended credential → S3A per-bucket Hadoop mapping.
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/vended/GcsVendedCredentialHadoopMapper.java	Implements GCS mapping for non-secret Hadoop connector keys (project/service host), leaving OAuth token in the serialized blob only.
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/vended/AdlsVendedCredentialHadoopMapper.java	Implements ADLS mapping for SAS/shared-key related ABFS Hadoop keys plus required non-secret auth-type companion property for SAS.
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/vended/OssVendedCredentialHadoopMapper.java	Implements OSS mapping for endpoint + access-key/secret/token Hadoop keys.
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/IcebergVendedCredentialUtil.java	Refactors credential propagation/extraction to use the new mapper abstraction and generalized credential extraction from FileIO properties.
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java	Extends hive.conf.hidden.list with new Iceberg FileIO credential key prefixes for GCS/ADLS/OSS.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[sonarqubecloud]

Quality Gate passed

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.6% Duplication on New Code

See analysis details on SonarQube Cloud