Allow jinja templating connection ids for all third party operators

Description

We use private staging and prod S3s(Ceph clusters for example) in our office.
So there are often cases where DAGs are running with only connection ids changed.
We prefer to use Param rather than to use hardcoded connection ids to make our code reusable.
I only gave an example for Amazon operator, but templating connection ids is required for other operators too.

Why is it needed?
Code reusability

Use case/motivation

with DAG(
    dag_id="example_s3",
    params={
        "aws_conn_id": Param("", type="string"),
    },
    ...
) as dag:

    create_object = S3CreateObjectOperator(
        task_id="create_object",
        s3_bucket=bucket_name,
        s3_key=key,
        data=DATA,
        replace=True,
        aws_conn_id="{{ params.aws_conn_id }}", # Params enable us to provide runtime configuration
    )

Related issues

No response

Are you willing to submit a PR?

Yes I am willing to submit a PR!

Code of Conduct

I agree to follow this project's Code of Conduct

This might be blocked by #29069 .

In general you could extend S3CreateObjectOperator.template_fields by create custom Operator with required template_fields

from airflow.providers.amazon.aws.operators.s3 import S3CreateObjectOperator

class AwesomeS3CreateObjectOperator:
    template_fields: ("aws_conn_id", *S3CreateObjectOperator.template_fields)

I think it might be a good first issue, I also required to collect all connection ID from the existed operators and list them, so everyone could pick up and make the changes.

Mark as good first issue, so maybe someone could volunteer free time to find at least most of the non-templated connections ids.

I collected all third party operators which have connection IDs.

Third party operators which have connection IDs

[ ] Airbyte
[ ] Alibaba
[ ] Amazon
[ ] Apache Beam
[ ] Apache Cassandra
[ ] Apache Drill
[ ] Apache Druid
[ ] Apache Flink
[ ] Apache HDFS
[ ] Apache Hive
[ ] Apache Impala
[ ] Apache Kafka
[ ] Apache Kylin
[ ] Apache Livy
[ ] Apache Pig
[ ] Apache Pinot
[ ] Apache Spark
[ ] Apprise
[ ] ArangoDB
[ ] Asana
[ ] Atlassian Jira
[ ] Cloudant
[ ] CNCF Kubernetes
[ ] Cohere
[ ] Common IO
[ ] Common SQL
[ ] Databricks
[ ] Datadog
[ ] dbt Cloud
[ ] Dingding
[ ] Discord
[ ] Docker
[ ] Elasticsearch
[ ] Exasol
[ ] Facebook
[ ] File Transfer Protocol (FTP)
[ ] GitHub
[ ] Google
[ ] gRPC
[ ] Hashicorp
[ ] Hypertext Transfer Protocol (HTTP)
[ ] IBM Cloudant
[ ] Influx DB
[ ] Internet Message Access Protocol (IMAP)
[ ] Java Database Connectivity (JDBC)
[ ] Jenkins
[ ] Microsoft Azure
[ ] Microsoft SQL Server (MSSQL)
[ ] Microsoft PowerShell Remoting Protocol (PSRP)
[ ] Microsoft Windows Remote Management (WinRM)
[ ] MongoDB
[ ] MySQL
[ ] Neo4j
[ ] ODBC
[ ] OpenAI
[ ] OpenFaaS
[ ] OpenLineage
[ ] Open Search
[ ] Opsgenie
[ ] Oracle
[ ] Pagerduty
[ ] Papermill
[ ] PgVector
[ ] Pinecone
[ ] PostgreSQL
[ ] Presto
[ ] Qdrant
[ ] Redis
[ ] Salesforce
[ ] Samba
[ ] Segment
[ ] Sendgrid
[ ] SFTP
[ ] Slack
[ ] SMTP
[ ] Snowflake
[ ] SQLite
[ ] SSH
[ ] Tableau
[ ] Tabular
[ ] Telegram
[ ] Teradata
[ ] Trino
[ ] Vertica
[ ] Weaviate
[ ] Yandex
[ ] Zendesk

I think we would just add the connection_id paramter to template_fields for all existing providers operators? However, that would be cumbersome and how to apply that convention moving forward? I'd be happy to make the updates to the operators, but I'm concerned about inconsistency.

@o-nikolas

@o-nikolas Gentle ping when you get a chance! I have updated test_template_fields in test_s3.py as requested and all 79 unit tests passed locally via Breeze.

I'd like to take the HTTP slice of this issue.
I'll add http_conn_id to template_fields on HttpOperator and HttpSensor only, so connection IDs can be Jinja-templated.
This will be a partial fix and won't close the issue.

I'd like to take the SFTP slice of this issue as well.
I'll add ssh_conn_id to template_fields on SFTPOperator and sftp_conn_id on SFTPSensor, so those connection IDs can be Jinja-templated. SFTPOperator takes ssh_conn_id because SFTPHook inherits SSHHook.
I won't change other providers, and this PR should not close the issue.

Taking the Redis slice.

RedisPublishOperator got redis_conn_id in #72883, but RedisKeySensor and RedisPubSubSensor still don't list it, so a templated connection id reaches RedisHook unrendered. Those two are the only ones left in the provider.

Add redis_conn_id to RedisKeySensor and RedisPubSubSensor template_fi…

Taking the Telegram slice.

Neither TelegramOperator nor TelegramFileOperator lists telegram_conn_id in template_fields, so a templated connection id reaches TelegramHook unrendered. Both store it in __init__ and read self.telegram_conn_id inside execute(), so rendering it is what the hook ends up receiving.

Both declare template_fields as a plain literal tuple with no helper building it, so this is not the aws_template_fields() no-op case from #70638.

Opened as #73263.

Navigation Menu

Repository navigation

Allow jinja templating connection ids for all third party operators #35259

Description

Use case/motivation

Related issues

Are you willing to submit a PR?

Code of Conduct

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Allow jinja templating connection ids for all third party operators #35259

Description

Description

Use case/motivation

Related issues

Are you willing to submit a PR?

Code of Conduct

Activity

Taragolis commented on Oct 30, 2023

Taragolis commented on Feb 19, 2024

kimminw00 commented on Feb 29, 2024

geraj1010 commented on Aug 15, 2024

divijaksolves commented on Jul 30, 2026

ucaeon commented on Sep 6, 2026

ucaeon commented on Sep 9, 2026

ShousenZHANG commented on Sep 12, 2026

ShousenZHANG commented on Sep 17, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions