Repository navigation
AIP-82: Add common messaging system providers #52712
Description
Body
related: #49938
Why
Several users have pointed out that the Event-Driven Scheduling feature introduced in Airflow 3 currently supports only Amazon SQS.
So I add Kafka common message queue provider in #49938, but there are still quite a few common providers that should be useful for airflow users to implement Event-Driven workflows.
What
Here are some common messaging systems:
- Amazon Kinesis Add Kinesis Data Streams trigger #71135
- Amazon MSK Add
MskHook#69000 - Amazon SQS
- Apache Kafka
- Azure Message Bus
- Trigger: Add Azure Service Bus Queue and Subscription triggers for async message processing #53356
- CommonMessageQueue Interface
- Google PubSub AIP-82: implement Google Pub/Sub message queue provider #54494
- Redis Add AwaitMessageTrigger for Redis PubSub #52917
reference for implementation: https://github.com/apache/airflow/pull/49938/files
Committer
- I acknowledge that I am a maintainer/committer of the Apache Airflow project.
Reacted by Kevin Yang and Kevin Liu
Activity
400Ping commentedon Jul 2, 2025
Hi, may I try the pubsub one?
Jasperora commentedon Jul 2, 2025
Hi, I would like to try the Redis one. Thanks!
prdai commentedon Jul 2, 2025
hi, could I implement the "Azure Message Bus" one? thanks :)
33 remaining items
potiuk commentedon Mar 2, 2026
@prdai We are unassigning you from this issue as part of our updated assignment policy.
This is not meant to discourage your contribution — quite the opposite! You are still very welcome to work on this issue and submit a PR for it. Simply comment that you are working on it and open a PR when ready.
We found that formal assignments were not working well, as they often prevented others from contributing when the assignee was not actively working on the issue.
potiuk commentedon Mar 2, 2026
@arnoldmr01 We are unassigning you from this issue as part of our updated assignment policy.
This is not meant to discourage your contribution — quite the opposite! You are still very welcome to work on this issue and submit a PR for it. Simply comment that you are working on it and open a PR when ready.
We found that formal assignments were not working well, as they often prevented others from contributing when the assignee was not actively working on the issue.
potiuk commentedon Mar 2, 2026
@Jasperora We are unassigning you from this issue as part of our updated assignment policy.
This is not meant to discourage your contribution — quite the opposite! You are still very welcome to work on this issue and submit a PR for it. Simply comment that you are working on it and open a PR when ready.
We found that formal assignments were not working well, as they often prevented others from contributing when the assignee was not actively working on the issue.
potiuk commentedon Mar 2, 2026
@SZL741023 We are unassigning you from this issue as part of our updated assignment policy.
This is not meant to discourage your contribution — quite the opposite! You are still very welcome to work on this issue and submit a PR for it. Simply comment that you are working on it and open a PR when ready.
We found that formal assignments were not working well, as they often prevented others from contributing when the assignee was not actively working on the issue.
Azure Message Bus - CommonMessageQueue Interface (apache#52712) (apac…
Azure Message Bus - CommonMessageQueue Interface (apache#52712) (apac…
Azure Message Bus - CommonMessageQueue Interface (apache#52712) (apac…
Azure Message Bus - CommonMessageQueue Interface (apache#52712) (apac…
aaron-y-chen commentedon Aug 5, 2026
I'm picking up the Kinesis work since I couldn't find an existing implementation PR.
I plan to split it into two PRs, following the Azure Service Bus approach:
jason810496 commentedon Aug 5, 2026
Sure! Big thanks for following-up on this feature.
aaron-y-chen commentedon Sep 22, 2026
The common-messaging integration for Kinesis is in #73509.
While working on it, I found bugs in the docstrings of all common messaging system providers. I opened a simple PR #73588 to fix them.
kaxil commentedon Oct 1, 2026
All the items tracked here are now closed or merged. Can this be closed, or is there remaining work it should track?