Microsoft Azure provider: Graph middleware (redirects, retries) silently skipped with microsoft-kiota-http >= 1.13 #74089

Under which category would you file this issue?

Providers

Apache Airflow version

3.3.2 (installed with the constraints-3-3 constraints)

What happened and how to reproduce it?

With microsoft-kiota-http >= 1.13 (the constraints-3-3 branch pins 1.14.0, constraints-main 1.14.1), every request made through KiotaRequestAdapterHook bypasses the whole Microsoft Graph middleware chain. The frozen constraints-3.3.2 still pins 1.12.3 and is not affected.

Cause: microsoft-kiota-http 1.13.0 stopped setting request.options (setattr(request, "options", request_options) in 1.12.x) and passes the request options in request.extensions[REQUEST_OPTIONS_KEY] instead. msgraph-core's AsyncGraphTransport.handle_async_request (1.5.1, the latest release) only runs its middleware pipeline when hasattr(request, "options"):

async def handle_async_request(self, request: httpx.Request) -> httpx.Response:
    if self.pipeline and hasattr(request, 'options'):
        self.set_request_context_and_feature_usage(request)
        response = await self.pipeline.send(request)
        return response

    response = await self.transport.handle_async_request(request)
    return response

So the request goes straight to the httpx transport, without the RedirectHandler, RetryHandler, ParametersNameDecodingHandler, UrlReplaceHandler, UserAgentHandler, HeadersInspectionHandler or GraphTelemetryHandler.

Effects:

• Redirects are not followed. Downloading a drive item (drives/{drive-id}/items/{item-id}/content, which answers 302 to a pre-authenticated SharePoint URL) returns Graph's raw 302 with an empty body: KiotaRequestAdapterHook.run(url=..., response_type="bytes") returns None.
• 429 and 503 with Retry-After are no longer retried by Kiota's RetryHandler.
• Calls answering 200 directly keep working, which makes the problem easy to miss. This affects KiotaRequestAdapterHook, MSGraphAsyncOperator, MSGraphSensor and the Power BI hooks/operators built on them.

Reproduce (no credentials needed; httpx.MockTransport plays Graph and SharePoint):

import asyncio

import httpx
from kiota_http.middleware import REQUEST_OPTIONS_KEY  # exists since microsoft-kiota-http 1.13
from msgraph_core import GraphClientFactory


def answer(request: httpx.Request) -> httpx.Response:
    if request.url.host == "graph.microsoft.com":
        return httpx.Response(302, headers={"location": "https://contoso.sharepoint.com/download.aspx"})
    return httpx.Response(200, content=b"%PDF-1.7")


async def main():
    client = GraphClientFactory.create_with_default_middleware(
        client=httpx.AsyncClient(transport=httpx.MockTransport(answer))
    )
    async with client:
        # How microsoft-kiota-http >= 1.13 builds the request: options in extensions only.
        request = httpx.Request(
            "GET",
            "https://graph.microsoft.com/v1.0/drives/d/items/i/content",
            extensions={REQUEST_OPTIONS_KEY: {}},
        )
        response = await client.send(request)
        print(response.status_code)  # 302: the RedirectHandler never ran


asyncio.run(main())

With microsoft-kiota-http==1.14.0 and msgraph-core==1.5.1 this prints 302. With the request options also set as request.options, which is what kiota-http 1.12.x did, the redirect is followed and it prints 200.

What you think should happen instead?

The Graph middleware should run whatever microsoft-kiota-http 1.x version the provider's dependency range allows (microsoft-kiota-http>=1.9.4,<2.0.0): redirects followed, 429/503 retried.

Until msgraph-core reads the options from request.extensions, the provider could either:

1. cap microsoft-kiota-http<1.13 (and keep the other microsoft-kiota-* packages on the matching 1.12.x), or
2. hand the options back to the transport in KiotaRequestAdapterHook, e.g. wrap the transport so that a request without options but with REQUEST_OPTIONS_KEY in extensions gets request.options = request.extensions[REQUEST_OPTIONS_KEY] before AsyncGraphTransport checks for it.

Option 2 keeps the newer Kiota releases usable. We run it as a plugin patch, and it is verified against a real 302 → 200 redirect. The root cause also deserves an issue on microsoftgraph/msgraph-sdk-python-core.

Operating System

Fedora 44 (container image)

Deployment

Other Docker-based deployment

Apache Airflow Provider(s)

microsoft-azure

Versions of Apache Airflow Providers

apache-airflow-providers-microsoft-azure==15.1.0
microsoft-kiota-http==1.14.0
microsoft-kiota-abstractions==1.14.0
microsoft-kiota-authentication-azure==1.14.0
msgraph-core==1.5.1
httpx==0.28.1

Anything else?

It happens on every request, not intermittently. Unit tests do not catch it because they mock the request adapter instead of sending a request through AsyncGraphTransport. A regression test should send a 302 through GraphClientFactory.create_with_default_middleware with an httpx.MockTransport, as in the reproduction above.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct