[pankajastro]

DataFusion credential resolution only handled AWS (and, in a sibling PR, GCS), so a wasb-backed DataSourceConfig failed with Unknown connection type wasb at registration, even though DataFusion's storage layer supports Azure Blob Storage.

Adds:

• StorageType.AZURE and az:// URI recognition in DataSourceConfig
• AzureObjectStorageProvider using DataFusion's MicrosoftAzure object store binding
• A wasb branch in _get_credentials resolving, in order: Azure AD service principal (tenant_id extra + login/password as client_id/client_secret), SAS token (sas_token extra, parsed into query pairs), then shared key (password, or shared_access_key/account_key extra) — falling back to ambient/environment-based auth when none apply
• Docs updated to mention Azure alongside S3

On the service-principal fields: DataFusion's MicrosoftAzure binding (crates/core/src/store.rs) panics if client_id/client_secret/tenant_id are only partially set — verified directly:

thread '<unnamed>' panicked at src/store.rs:114:17:
client_id, client_secret, tenat_id must be all set or all None

(PyO3 converts this to a PanicException rather than crashing the process, but it's still an ugly failure mode.) So this PR only forwards all three when the connection genuinely has all three (tenant_id extra plus both login and password); otherwise it falls through to the next auth mode rather than risking a partial combination.

Unsupported fields: connection_string, managed_identity_client_id, workload_identity_tenant_id, and a URL-form sas_token aren't supported by DataFusion's Azure binding. Setting any of these now raises a clear error instead of silently falling back to a different identity.

Scope: only az:// URIs are recognized for now — abfs:///abfss:// are not. Those schemes embed the storage account in the URI itself (e.g. abfss://container@account.dfs.core.windows.net/path), which is a different shape from az:// and raises a real question given host now takes priority when resolving the account from the connection: if the URI's account differs from the connection's, which one wins? That's a design decision worth its own PR rather than bundling in here.

---

Was generative AI tooling used to co-author this PR?

• Yes — Claude Code (Sonnet 5)

Generated-by: Claude Code (Sonnet 5) following the guidelines

[kaxil]

Round 2 at 016a9bc. The three code comments from the first round are addressed: the account now comes from host first, the Azure error message uses .value, and the urllib.parse import is at module level. Keeping abfs:// and abfss:// for a separate PR is fine.

Four new items, all measured against the real datafusion binding and object_store source:

• The worker's AZURE_* environment is read by from_env() before the connection's credential is overlaid, and object_store checks an environment access key or workload identity before a SAS token or client secret. With AZURE_STORAGE_ACCOUNT_KEY set, or the AKS workload-identity variables set, the connection's SAS or service principal is never used. The docs currently say ambient credentials come last.
• _resolve_wasb_account keeps only the first host label and the binding has no endpoint parameter, so sovereign-cloud, private-endpoint and Azurite hosts are silently redirected to *.blob.core.windows.net.
• A partial service principal (tenant_id with login or password missing) falls through to ambient auth or forwards the client secret as a shared key instead of raising.
• With neither host nor login set, the account becomes the string "None", where the previous code let the binding fall back to AZURE_STORAGE_ACCOUNT_NAME. That is the shape of the default wasb_default connection the example DAG uses.

Inline comments have the details and suggested fixes. Verdict: hold until these are addressed.

[kaxil]

@pankajastro Conflicts !

[pankajastro]

@pankajastro Conflicts !

Rebased onto current main, thanks — should be conflict-free now.