Repository navigation
Add get_async_extra_dejson to common.compat connection helpers - #74147
Merged
dabla merged 1 commit intoOct 3, 2026
Merged
Conversation
Connection.extra_dejson masks the extra's secrets with a synchronous call to the supervisor, which raises DeadlockImminentError when a hook reads it on an event loop with another async call in flight. Airflow 3.3.2+ has Connection.aextra_dejson() (apache#71890), but providers that still support older versions cannot call it directly, so async hooks fall back to json.loads(conn.extra) and skip the masking (apache#72130). get_async_extra_dejson(conn) awaits Connection.aextra_dejson() when it exists, and otherwise runs extra_dejson in a worker thread, where blocking on the supervisor is safe, as get_async_connection() does for get_connection(). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 3, 2026
shahar1
approved these changes
Oct 3, 2026
1 task done
dabla
added a commit
to dabla/airflow
that referenced
this pull request
Oct 3, 2026
Replace the local aextra_dejson fallback with the common.compat helper added in apache#74147, which masks the extra's secrets asynchronously on Airflow 3.3.2+ and reads extra_dejson from a worker thread on older versions. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
dabla
added a commit
to dabla/airflow
that referenced
this pull request
Oct 3, 2026
Replace the local aextra_dejson fallback with the common.compat helper added in apache#74147, which masks the extra's secrets asynchronously on Airflow 3.3.2+ and reads extra_dejson from a worker thread on older versions. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 3, 2026
1 task done
dabla
added a commit
to dabla/airflow
that referenced
this pull request
Oct 12, 2026
Replace the local aextra_dejson fallback with the common.compat helper added in apache#74147, which masks the extra's secrets asynchronously on Airflow 3.3.2+ and reads extra_dejson from a worker thread on older versions. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 14, 2026
dabla
added a commit
to dabla/airflow
that referenced
this pull request
Oct 14, 2026
Replace the local aextra_dejson fallback with the common.compat helper added in apache#74147, which masks the extra's secrets asynchronously on Airflow 3.3.2+ and reads extra_dejson from a worker thread on older versions. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
dabla
added a commit
that referenced
this pull request
Oct 16, 2026
* Add async aget_conn and acreate_embeddings to OpenAIHook get_conn() resolves the connection with get_connection() and reads extra_dejson, whose secret masking sends to the supervisor synchronously; inside an async task with another async SDK call in flight both raise DeadlockImminentError. aget_conn() returns an AsyncOpenAI client configured exactly like get_conn()'s, from get_async_connection() with the extra from Connection.aextra_dejson() (#71890, Airflow 3.3.2+), or the raw extra via json.loads on older versions (as in #72130). The client settings move into _client_kwargs(), shared by both paths. acreate_embeddings() is the async counterpart of create_embeddings(). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Use get_async_extra_dejson from common.compat Replace the local aextra_dejson fallback with the common.compat helper added in #74147, which masks the extra's secrets asynchronously on Airflow 3.3.2+ and reads extra_dejson from a worker thread on older versions. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Spec the mocked AsyncOpenAI client in test_acreate_embeddings MagicMock() without a spec accepts any attribute, so a typo on the async path such as client.embedding would still pass. spec=AsyncOpenAI guards the client's namespace the way the sync tests' spec=OpenAI does. * Assert aget_conn reads the extra through aextra_dejson The connection mock carried an extra string and no test checked that aextra_dejson() was awaited, so a hook reading json.loads(conn.extra), which skips the secret masking, kept every test green. Drop extra from the mock's spec and assert the await in the tests that have it. * Share the embeddings result shaping between the sync and async paths acreate_embeddings copied the three lines that pick one vector for a single input and sort one vector per item for a batch. Move them into _embeddings_result, used by create_embeddings and acreate_embeddings, so the two paths cannot drift, like _client_kwargs for the client. * Resolve the async connection through the hook get_conn resolves through self.get_connection, so a subclass override applies there, but aget_conn called get_async_connection without hook= and the lookup went through BaseHook. Pass hook=self, as AnthropicHook.get_async_conn does, and test it with a subclass that overrides aget_connection, through the real common.compat helper. * Build the async client per call instead of caching it on the hook _aget_cached_conn checked the cache before two awaits, so concurrent calls on a fresh hook each built a client, dropped all but one unclosed, and the kept one was never closed and stayed bound to the first event loop. aget_conn now builds a client per call and says to close it, as AnthropicHook.get_async_conn does; acreate_embeddings closes its client with async with. Tested with three concurrent calls on one hook. * State that aget_conn avoids synchronous Task SDK calls from Airflow 3.3.2 Before 3.3.2, core's aget_connection still masks the connection's secrets synchronously, so the docstring promised more than older cores give. * Pin the per-call client in the concurrent embeddings test Patching aget_conn let the test supply distinct clients itself, so it kept passing if aget_conn cached one client on the hook again. * Note the token exchange per client for workload identity in aget_conn The SDK caches the exchanged token on each client, so callers making many calls should know they can keep one client open and reuse it. * Require common.compat 1.21.0 for get_async_extra_dejson The hook imports get_async_extra_dejson at module level, so with an older common.compat the whole hook, sync get_conn included, fails to import. * Restore the use-next-version marker on the common.compat dependency Only release managers raise provider dependency floors; the marker lets them bump it to the release that ships get_async_extra_dejson. * Build the embeddings usage as a Usage model in the async test client mypy checks the annotated _async_client helper and rejects the dict passed to CreateEmbeddingResponse.usage, which is typed Usage. * Update providers/openai/src/airflow/providers/openai/hooks/openai.py Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com> * Update providers/openai/src/airflow/providers/openai/hooks/openai.py Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com> * Update providers/openai/src/airflow/providers/openai/hooks/openai.py Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com> * Update providers/openai/tests/unit/openai/hooks/test_openai.py Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com> * Drop test docstrings that repeat the test names --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com>
dabla
added a commit
to dabla/airflow
that referenced
this pull request
Oct 16, 2026
Replace the local aextra_dejson fallback with the common.compat helper added in apache#74147, which masks the extra's secrets asynchronously on Airflow 3.3.2+ and reads extra_dejson from a worker thread on older versions. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 task done
dabla
added a commit
to dabla/airflow
that referenced
this pull request
Oct 18, 2026
get_async_extra_dejson (apache#74147) keeps the extra's secret masking off the event loop on every version, but fetching the connection did not: before 3.3.2, BaseHook.aget_connection masks synchronously in core, so async hooks could still raise DeadlockImminentError on 3.1 to 3.3.1.
10 of 93 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Connection.extra_dejsonmasks the extra's secrets with a synchronous call to the supervisor. When a hook reads it on an event loop with another async call in flight (an async task running items concurrently, a trigger), that raisesDeadlockImminentError.Airflow 3.3.2+ has
Connection.aextra_dejson()(#71890, backported in #72895), but providers that still support older Airflow versions can't call it directly. Async hooks therefore deserialize the raw extra themselves (json.loads(conn.extra), e.g. #72130 and the msgraph hook), which skips masking the individual values of the extra.This adds
get_async_extra_dejson(conn)next toget_async_connection()inairflow.providers.common.compat.connection:await conn.aextra_dejson()conn.extra_dejsonin a worker thread (asgiref.sync_to_async), where blocking on the supervisor is safe — the same fallbackget_async_connection()uses forget_connection()Used by these PRs, which are drafts until this one is merged:
aget_conn()/acreate_embeddings()forOpenAIHookaget_conn()/acreate_agent()forPydanticAIHookKiotaRequestAdapterHook(msgraph) reads its extra with it instead ofjson.loads(connection.extra)Tests cover the
aextra_dejson()path (whileextra_dejsonraises), the fallback running off the event-loop thread, and a realConnectionwith and without an extra.Was generative AI tooling used to co-author this PR?
Generated-by: Claude Opus 5.5 following the guidelines