Skip to content

sqlalchemy.exc.PendingRollbackError: Can't reconnect until invalid transaction is rolled back #57470

Closed
#57815
Closed
sqlalchemy.exc.PendingRollbackError: Can't reconnect until invalid transaction is rolled back#57470
#57815
Labels
affected_version:3.1Issues Reported for 3.1area:corekind:bugThis is a clearly a bugneeds-triagelabel for new issues that we didn't triage yetpriority:upgrade_to_airflow3

Description

@rebel-wansoo
rebel-wansoo
opened on Oct 29, 2025

Apache Airflow version

3.1.1

If "Other Airflow 2/3 version" selected, which one?

No response

What happened?

After upgrading from Airflow 3.1.0 to 3.1.1, the following error started occurring on the API server:
sqlalchemy.exc.PendingRollbackError: Can't reconnect until invalid transaction is rolled back.

In addition, when accessing the UI, an Internal Server Error appears in the browser, and in some cases, the Oops page is displayed instead.
This issue occurs very frequently, though there are occasional cases where it does not occur.

What you think should happen instead?

No response

How to reproduce

After upgrading from Airflow 3.1.0 to Airflow 3.1.1, please log in.

Operating System

ubuntu

Versions of Apache Airflow Providers

No response

Deployment

Official Apache Airflow Helm Chart

Deployment details

No response

Anything else?

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

Activity

  1. rebel-wansoo
    added
    kind:bugThis is a clearly a bug
    area:core
    needs-triagelabel for new issues that we didn't triage yet
    on Oct 29, 2025
  2. dosubot
    added
    area:APIAirflow's REST/HTTP API
    on Oct 29, 2025

  3. arimus commented on Oct 29, 2025

    @arimus
    arimus
    on Oct 29, 2025

    We are having the same issue after an upgrade yesterday. It's preventing even logging in to our instance

  4. vikramkoka
    added
    affected_version:3.1Issues Reported for 3.1
    priority:upgrade_to_airflow3
    and removed
    area:APIAirflow's REST/HTTP API
    on Oct 29, 2025

  5. vikramkoka commented on Oct 29, 2025

    @vikramkoka
    vikramkoka
    on Oct 29, 2025
    Contributor

    This seems very strange. Can you please share any details around this?

  6. dougforpres commented on Oct 29, 2025

    @dougforpres
    dougforpres
    on Oct 29, 2025

    @vikramkoka - I'm working with @arimus and can explain what we're seeing.

    After the upgrade last-night things started acting weird.
    It all seemed to be ok until 15:00 (server time) today. That's when I returned to the app after leaving it apparently "ok" last night.

    When I click log-out I will sometimes get a 500 error with a huge stacktrace in the log indicating a transaction needs to be rolled back.
    When I am able to log out, and I try to log in, sometimes I'll get a generic "Server Error" page, and other times the "Ooops" page will appear.
    When I am able to log on, I'll get the home page with errors on it indicating I'm not logged on, then the page will refresh and appear to be ok.

    Here is a log from the start of the 3.1.1 server yesterday.
    You can see it all seemed to be ok until 15:00 today (the 29th) when a page request got a mysql timeout... then, when the user (me, as it happens) tried to log on the errors about transactions needing to roll-back started.
    Apologies if this blob is too big

    Oct 28 16:39:08 integration-airflow-new.mht python3.13[859034]: INFO:     Uvicorn running on http://0.0.0.0:8080 (Press CTRL+C to quit)
Oct 28 16:39:08 integration-airflow-new.mht python3.13[859034]: INFO:     Started parent process [859034]
Oct 28 16:39:08 integration-airflow-new.mht python3.13[859034]:   ____________       _____________
Oct 28 16:39:08 integration-airflow-new.mht python3.13[859034]:  ____    |__( )_________  __/__  /________      __
Oct 28 16:39:08 integration-airflow-new.mht python3.13[859034]: ____  /| |_  /__  ___/_  /_ __  /_  __ \_ | /| / /
Oct 28 16:39:08 integration-airflow-new.mht python3.13[859034]: ___  ___ |  / _  /   _  __/ _  / / /_/ /_ |/ |/ /
Oct 28 16:39:08 integration-airflow-new.mht python3.13[859034]:  _/_/  |_/_/  /_/    /_/    /_/  \____/____/|__/
Oct 28 16:39:12 integration-airflow-new.mht python3.13[859126]: /opt/airflow/.local/lib/python3.13/site-packages/airflow/api_fastapi/execution_api/routes/__init__.py:23 DeprecationWarning: 'HTTP_422_UNPROCESSABLE_ENTITY' is deprecated. Use 'HTTP_422_UNPROCESSABLE_CONTENT' instead.
Oct 28 16:39:12 integration-airflow-new.mht python3.13[859124]: /opt/airflow/.local/lib/python3.13/site-packages/airflow/api_fastapi/execution_api/routes/__init__.py:23 DeprecationWarning: 'HTTP_422_UNPROCESSABLE_ENTITY' is deprecated. Use 'HTTP_422_UNPROCESSABLE_CONTENT' instead.
Oct 28 16:39:13 integration-airflow-new.mht python3.13[859126]: [2025-10-28T16:39:13.072189Z] {workday.py:41} WARNING - Could not import pandas. Holidays will not be considered.
Oct 28 16:39:13 integration-airflow-new.mht python3.13[859127]: /opt/airflow/.local/lib/python3.13/site-packages/airflow/api_fastapi/execution_api/routes/__init__.py:23 DeprecationWarning: 'HTTP_422_UNPROCESSABLE_ENTITY' is deprecated. Use 'HTTP_422_UNPROCESSABLE_CONTENT' instead.
Oct 28 16:39:13 integration-airflow-new.mht python3.13[859125]: /opt/airflow/.local/lib/python3.13/site-packages/airflow/api_fastapi/execution_api/routes/__init__.py:23 DeprecationWarning: 'HTTP_422_UNPROCESSABLE_ENTITY' is deprecated. Use 'HTTP_422_UNPROCESSABLE_CONTENT' instead.
Oct 28 16:39:13 integration-airflow-new.mht python3.13[859126]: /opt/airflow/.local/lib/python3.13/site-packages/airflow/providers/fab/auth_manager/fab_auth_manager.py:108 RemovedInAirflow4Warning: The airflow.security.permissions module is deprecated; please see https://airflow.apache.org/docs/apache-airflow/stable/security/deprecated_permissions.html
Oct 28 16:39:13 integration-airflow-new.mht python3.13[859126]: /opt/airflow/.local/lib/python3.13/site-packages/flask_limiter/extension.py:324 UserWarning: Using the in-memory storage for tracking rate limits as no storage was explicitly specified. This is not recommended for production use. See: https://flask-limiter.readthedocs.io#configuring-a-storage-backend for documentation about configuring the storage backend.
Oct 28 16:39:13 integration-airflow-new.mht python3.13[859124]: [2025-10-28T16:39:13.553595Z] {workday.py:41} WARNING - Could not import pandas. Holidays will not be considered.
Oct 28 16:39:13 integration-airflow-new.mht python3.13[859127]: [2025-10-28T16:39:13.734821Z] {workday.py:41} WARNING - Could not import pandas. Holidays will not be considered.
Oct 28 16:39:13 integration-airflow-new.mht python3.13[859126]: appbuilder.app is deprecated and will be removed in a future version. Use current_app instead
Oct 28 16:39:13 integration-airflow-new.mht python3.13[859126]: [2025-10-28T16:39:13.809361Z] {init_appbuilder.py:228} WARNING - appbuilder.app is deprecated and will be removed in a future version. Use current_app instead
Oct 28 16:39:13 integration-airflow-new.mht python3.13[859124]: /opt/airflow/.local/lib/python3.13/site-packages/airflow/providers/fab/auth_manager/fab_auth_manager.py:108 RemovedInAirflow4Warning: The airflow.security.permissions module is deprecated; please see https://airflow.apache.org/docs/apache-airflow/stable/security/deprecated_permissions.html
Oct 28 16:39:13 integration-airflow-new.mht python3.13[859125]: [2025-10-28T16:39:13.855514Z] {workday.py:41} WARNING - Could not import pandas. Holidays will not be considered.
Oct 28 16:39:13 integration-airflow-new.mht python3.13[859126]: appbuilder.app is deprecated and will be removed in a future version. Use current_app instead
Oct 28 16:39:13 integration-airflow-new.mht python3.13[859126]: [2025-10-28T16:39:13.867368Z] {init_appbuilder.py:228} WARNING - appbuilder.app is deprecated and will be removed in a future version. Use current_app instead
Oct 28 16:39:13 integration-airflow-new.mht python3.13[859126]: appbuilder.app is deprecated and will be removed in a future version. Use current_app instead
Oct 28 16:39:13 integration-airflow-new.mht python3.13[859126]: [2025-10-28T16:39:13.908115Z] {init_appbuilder.py:228} WARNING - appbuilder.app is deprecated and will be removed in a future version. Use current_app instead
Oct 28 16:39:13 integration-airflow-new.mht python3.13[859126]: appbuilder.app is deprecated and will be removed in a future version. Use current_app instead
Oct 28 16:39:13 integration-airflow-new.mht python3.13[859126]: [2025-10-28T16:39:13.963824Z] {init_appbuilder.py:228} WARNING - appbuilder.app is deprecated and will be removed in a future version. Use current_app instead
Oct 28 16:39:13 integration-airflow-new.mht python3.13[859124]: /opt/airflow/.local/lib/python3.13/site-packages/flask_limiter/extension.py:324 UserWarning: Using the in-memory storage for tracking rate limits as no storage was explicitly specified. This is not recommended for production use. See: https://flask-limiter.readthedocs.io#configuring-a-storage-backend for documentation about configuring the storage backend.
Oct 28 16:39:14 integration-airflow-new.mht python3.13[859127]: /opt/airflow/.local/lib/python3.13/site-packages/airflow/providers/fab/auth_manager/fab_auth_manager.py:108 RemovedInAirflow4Warning: The airflow.security.permissions module is deprecated; please see https://airflow.apache.org/docs/apache-airflow/stable/security/deprecated_permissions.html
Oct 28 16:39:14 integration-airflow-new.mht python3.13[859126]: appbuilder.app is deprecated and will be removed in a future version. Use current_app instead
Oct 28 16:39:14 integration-airflow-new.mht python3.13[859126]: [2025-10-28T16:39:14.043607Z] {init_appbuilder.py:228} WARNING - appbuilder.app is deprecated and will be removed in a future version. Use current_app instead
Oct 28 16:39:14 integration-airflow-new.mht python3.13[859126]: appbuilder.app is deprecated and will be removed in a future version. Use current_app instead
Oct 28 16:39:14 integration-airflow-new.mht python3.13[859126]: [2025-10-28T16:39:14.096765Z] {init_appbuilder.py:228} WARNING - appbuilder.app is deprecated and will be removed in a future version. Use current_app instead
Oct 28 16:39:14 integration-airflow-new.mht python3.13[859127]: /opt/airflow/.local/lib/python3.13/site-packages/flask_limiter/extension.py:324 UserWarning: Using the in-memory storage for tracking rate limits as no storage was explicitly specified. This is not recommended for production use. See: https://flask-limiter.readthedocs.io#configuring-a-storage-backend for documentation about configuring the storage backend.
Oct 28 16:39:14 integration-airflow-new.mht python3.13[859125]: /opt/airflow/.local/lib/python3.13/site-packages/airflow/providers/fab/auth_manager/fab_auth_manager.py:108 RemovedInAirflow4Warning: The airflow.security.permissions module is deprecated; please see https://airflow.apache.org/docs/apache-airflow/stable/security/deprecated_permissions.html
Oct 28 16:39:14 integration-airflow-new.mht python3.13[859125]: /opt/airflow/.local/lib/python3.13/site-packages/flask_limiter/extension.py:324 UserWarning: Using the in-memory storage for tracking rate limits as no storage was explicitly specified. This is not recommended for production use. See: https://flask-limiter.readthedocs.io#configuring-a-storage-backend for documentation about configuring the storage backend.
Oct 28 16:39:14 integration-airflow-new.mht python3.13[859126]: /opt/airflow/.local/lib/python3.13/site-packages/airflow/providers/fab/www/extensions/init_jinja_globals.py:43 DeprecationWarning: The expose_hostname option in [webserver] has been moved to the expose_hostname option in [fab] - the old setting has been used, but please update your config.
Oct 28 16:39:14 integration-airflow-new.mht python3.13[859124]: appbuilder.app is deprecated and will be removed in a future version. Use current_app instead
Oct 28 16:39:14 integration-airflow-new.mht python3.13[859124]: [2025-10-28T16:39:14.300669Z] {init_appbuilder.py:228} WARNING - appbuilder.app is deprecated and will be removed in a future version. Use current_app instead
Oct 28 16:39:14 integration-airflow-new.mht python3.13[859124]: appbuilder.app is deprecated and will be removed in a future version. Use current_app instead
Oct 28 16:39:14 integration-airflow-new.mht python3.13[859124]: [2025-10-28T16:39:14.359324Z] {init_appbuilder.py:228} WARNING - appbuilder.app is deprecated and will be removed in a future version. Use current_app instead
Oct 28 16:39:14 integration-airflow-new.mht python3.13[859124]: appbuilder.app is deprecated and will be removed in a future version. Use current_app instead
Oct 28 16:39:14 integration-airflow-new.mht python3.13[859124]: [2025-10-28T16:39:14.399801Z] {init_appbuilder.py:228} WARNING - appbuilder.app is deprecated and will be removed in a future version. Use current_app instead
Oct 28 16:39:14 integration-airflow-new.mht python3.13[859127]: appbuilder.app is deprecated and will be removed in a future version. Use current_app instead
Oct 28 16:39:14 integration-airflow-new.mht python3.13[859127]: [2025-10-28T16:39:14.402973Z] {init_appbuilder.py:228} WARNING - appbuilder.app is deprecated and will be removed in a future version. Use current_app instead
Oct 28 16:39:14 integration-airflow-new.mht python3.13[859124]: appbuilder.app is deprecated and will be removed in a future version. Use current_app instead
Oct 28 16:39:14 integration-airflow-new.mht python3.13[859124]: [2025-10-28T16:39:14.442189Z] {init_appbuilder.py:228} WARNING - appbuilder.app is deprecated and will be removed in a future version. Use current_app instead
Oct 28 16:39:14 integration-airflow-new.mht python3.13[859127]: appbuilder.app is deprecated and will be removed in a future version. Use current_app instead
Oct 28 16:39:14 integration-airflow-new.mht python3.13[859127]: [2025-10-28T16:39:14.452262Z] {init_appbuilder.py:228} WARNING - appbuilder.app is deprecated and will be removed in a future version. Use current_app instead
Oct 28 16:39:14 integration-airflow-new.mht python3.13[859124]: appbuilder.app is deprecated and will be removed in a future version. Use current_app instead
Oct 28 16:39:14 integration-airflow-new.mht python3.13[859124]: [2025-10-28T16:39:14.481730Z] {init_appbuilder.py:228} WARNING - appbuilder.app is deprecated and will be removed in a future version. Use current_app instead
Oct 28 16:39:14 integration-airflow-new.mht python3.13[859127]: appbuilder.app is deprecated and will be removed in a future version. Use current_app instead
Oct 28 16:39:14 integration-airflow-new.mht python3.13[859127]: [2025-10-28T16:39:14.491940Z] {init_appbuilder.py:228} WARNING - appbuilder.app is deprecated and will be removed in a future version. Use current_app instead
Oct 28 16:39:14 integration-airflow-new.mht python3.13[859125]: appbuilder.app is deprecated and will be removed in a future version. Use current_app instead
Oct 28 16:39:14 integration-airflow-new.mht python3.13[859125]: [2025-10-28T16:39:14.495844Z] {init_appbuilder.py:228} WARNING - appbuilder.app is deprecated and will be removed in a future version. Use current_app instead
Oct 28 16:39:14 integration-airflow-new.mht python3.13[859124]: appbuilder.app is deprecated and will be removed in a future version. Use current_app instead
Oct 28 16:39:14 integration-airflow-new.mht python3.13[859124]: [2025-10-28T16:39:14.523574Z] {init_appbuilder.py:228} WARNING - appbuilder.app is deprecated and will be removed in a future version. Use current_app instead
Oct 28 16:39:14 integration-airflow-new.mht python3.13[859127]: appbuilder.app is deprecated and will be removed in a future version. Use current_app instead
Oct 28 16:39:14 integration-airflow-new.mht python3.13[859127]: [2025-10-28T16:39:14.542174Z] {init_appbuilder.py:228} WARNING - appbuilder.app is deprecated and will be removed in a future version. Use current_app instead
Oct 28 16:39:14 integration-airflow-new.mht python3.13[859125]: appbuilder.app is deprecated and will be removed in a future version. Use current_app instead
Oct 28 16:39:14 integration-airflow-new.mht python3.13[859125]: [2025-10-28T16:39:14.545462Z] {init_appbuilder.py:228} WARNING - appbuilder.app is deprecated and will be removed in a future version. Use current_app instead
Oct 28 16:39:14 integration-airflow-new.mht python3.13[859125]: appbuilder.app is deprecated and will be removed in a future version. Use current_app instead
Oct 28 16:39:14 integration-airflow-new.mht python3.13[859125]: [2025-10-28T16:39:14.585467Z] {init_appbuilder.py:228} WARNING - appbuilder.app is deprecated and will be removed in a future version. Use current_app instead
Oct 28 16:39:14 integration-airflow-new.mht python3.13[859127]: appbuilder.app is deprecated and will be removed in a future version. Use current_app instead
Oct 28 16:39:14 integration-airflow-new.mht python3.13[859127]: [2025-10-28T16:39:14.595321Z] {init_appbuilder.py:228} WARNING - appbuilder.app is deprecated and will be removed in a future version. Use current_app instead
Oct 28 16:39:14 integration-airflow-new.mht python3.13[859125]: appbuilder.app is deprecated and will be removed in a future version. Use current_app instead
Oct 28 16:39:14 integration-airflow-new.mht python3.13[859125]: [2025-10-28T16:39:14.626452Z] {init_appbuilder.py:228} WARNING - appbuilder.app is deprecated and will be removed in a future version. Use current_app instead
Oct 28 16:39:14 integration-airflow-new.mht python3.13[859127]: appbuilder.app is deprecated and will be removed in a future version. Use current_app instead
Oct 28 16:39:14 integration-airflow-new.mht python3.13[859127]: [2025-10-28T16:39:14.647988Z] {init_appbuilder.py:228} WARNING - appbuilder.app is deprecated and will be removed in a future version. Use current_app instead
Oct 28 16:39:14 integration-airflow-new.mht python3.13[859125]: appbuilder.app is deprecated and will be removed in a future version. Use current_app instead
Oct 28 16:39:14 integration-airflow-new.mht python3.13[859125]: [2025-10-28T16:39:14.667622Z] {init_appbuilder.py:228} WARNING - appbuilder.app is deprecated and will be removed in a future version. Use current_app instead
Oct 28 16:39:14 integration-airflow-new.mht python3.13[859124]: /opt/airflow/.local/lib/python3.13/site-packages/airflow/providers/fab/www/extensions/init_jinja_globals.py:43 DeprecationWarning: The expose_hostname option in [webserver] has been moved to the expose_hostname option in [fab] - the old setting has been used, but please update your config.
Oct 28 16:39:14 integration-airflow-new.mht python3.13[859125]: appbuilder.app is deprecated and will be removed in a future version. Use current_app instead
Oct 28 16:39:14 integration-airflow-new.mht python3.13[859125]: [2025-10-28T16:39:14.723481Z] {init_appbuilder.py:228} WARNING - appbuilder.app is deprecated and will be removed in a future version. Use current_app instead
Oct 28 16:39:14 integration-airflow-new.mht python3.13[859127]: /opt/airflow/.local/lib/python3.13/site-packages/airflow/providers/fab/www/extensions/init_jinja_globals.py:43 DeprecationWarning: The expose_hostname option in [webserver] has been moved to the expose_hostname option in [fab] - the old setting has been used, but please update your config.
Oct 28 16:39:14 integration-airflow-new.mht python3.13[859125]: /opt/airflow/.local/lib/python3.13/site-packages/airflow/providers/fab/www/extensions/init_jinja_globals.py:43 DeprecationWarning: The expose_hostname option in [webserver] has been moved to the expose_hostname option in [fab] - the old setting has been used, but please update your config.
Oct 28 16:39:15 integration-airflow-new.mht python3.13[859126]: /opt/airflow/.local/lib/python3.13/site-packages/airflow/api_fastapi/core_api/routes/public/__init__.py:36 DeprecationWarning: 'HTTP_422_UNPROCESSABLE_ENTITY' is deprecated. Use 'HTTP_422_UNPROCESSABLE_CONTENT' instead.
Oct 28 16:39:15 integration-airflow-new.mht python3.13[859124]: /opt/airflow/.local/lib/python3.13/site-packages/airflow/api_fastapi/core_api/routes/public/__init__.py:36 DeprecationWarning: 'HTTP_422_UNPROCESSABLE_ENTITY' is deprecated. Use 'HTTP_422_UNPROCESSABLE_CONTENT' instead.
Oct 28 16:39:15 integration-airflow-new.mht python3.13[859127]: /opt/airflow/.local/lib/python3.13/site-packages/airflow/api_fastapi/core_api/routes/public/__init__.py:36 DeprecationWarning: 'HTTP_422_UNPROCESSABLE_ENTITY' is deprecated. Use 'HTTP_422_UNPROCESSABLE_CONTENT' instead.
Oct 28 16:39:15 integration-airflow-new.mht python3.13[859125]: /opt/airflow/.local/lib/python3.13/site-packages/airflow/api_fastapi/core_api/routes/public/__init__.py:36 DeprecationWarning: 'HTTP_422_UNPROCESSABLE_ENTITY' is deprecated. Use 'HTTP_422_UNPROCESSABLE_CONTENT' instead.
Oct 28 16:39:18 integration-airflow-new.mht python3.13[859126]: INFO:     Started server process [859126]
Oct 28 16:39:18 integration-airflow-new.mht python3.13[859126]: INFO:     Waiting for application startup.
Oct 28 16:39:18 integration-airflow-new.mht python3.13[859126]: INFO:     Application startup complete.
Oct 28 16:39:18 integration-airflow-new.mht python3.13[859124]: INFO:     Started server process [859124]
Oct 28 16:39:18 integration-airflow-new.mht python3.13[859124]: INFO:     Waiting for application startup.
Oct 28 16:39:18 integration-airflow-new.mht python3.13[859124]: INFO:     Application startup complete.
Oct 28 16:39:19 integration-airflow-new.mht python3.13[859125]: INFO:     Started server process [859125]
Oct 28 16:39:19 integration-airflow-new.mht python3.13[859125]: INFO:     Waiting for application startup.
Oct 28 16:39:19 integration-airflow-new.mht python3.13[859125]: INFO:     Application startup complete.
Oct 28 16:39:19 integration-airflow-new.mht python3.13[859127]: INFO:     Started server process [859127]
Oct 28 16:39:19 integration-airflow-new.mht python3.13[859127]: INFO:     Waiting for application startup.
Oct 28 16:39:19 integration-airflow-new.mht python3.13[859127]: INFO:     Application startup complete.
Oct 28 16:39:24 integration-airflow-new.mht python3.13[859125]: INFO:     127.0.0.1:39622 - "GET /api/v2/monitor/health HTTP/1.1" 200 OK
Oct 28 16:39:24 integration-airflow-new.mht python3.13[859127]: INFO:     127.0.0.1:39590 - "GET /ui/dashboard/dag_stats HTTP/1.1" 200 OK
Oct 28 16:39:24 integration-airflow-new.mht python3.13[859127]: INFO:     127.0.0.1:39600 - "GET /api/v2/pools HTTP/1.1" 200 OK
Oct 28 16:39:24 integration-airflow-new.mht python3.13[859126]: INFO:     127.0.0.1:39616 - "GET /api/v2/dags/~/dagRuns/~/hitlDetails?state=deferred&response_received=false HTTP/1.1" 200 OK
Oct 28 16:39:26 integration-airflow-new.mht python3.13[859127]: INFO:     127.0.0.1:39630 - "GET /ui/dashboard/historical_metrics_data?start_date=2025-10-27T05%3A42%3A52.587Z HTTP/1.1" 200 OK
Oct 28 16:39:27 integration-airflow-new.mht python3.13[859127]: INFO:     127.0.0.1:39640 - "GET / HTTP/1.1" 200 OK
Oct 28 16:39:27 integration-airflow-new.mht python3.13[859127]: INFO:     127.0.0.1:39656 - "GET /static/assets/index-BjF9k7EU.js HTTP/1.1" 200 OK
Oct 28 16:39:28 integration-airflow-new.mht python3.13[859127]: INFO:     127.0.0.1:39670 - "GET /ui/config HTTP/1.1" 401 Unauthorized
Oct 28 16:39:28 integration-airflow-new.mht python3.13[859127]: INFO:     127.0.0.1:39716 - "GET /static/i18n/locales/en/browse.json HTTP/1.1" 200 OK
Oct 28 16:39:28 integration-airflow-new.mht python3.13[859125]: INFO:     127.0.0.1:39700 - "GET /static/i18n/locales/en/dashboard.json HTTP/1.1" 200 OK
Oct 28 16:39:28 integration-airflow-new.mht python3.13[859127]: INFO:     127.0.0.1:39686 - "GET /static/i18n/locales/en/dags.json HTTP/1.1" 200 OK
Oct 28 16:39:29 integration-airflow-new.mht python3.13[859127]: INFO:     127.0.0.1:39738 - "GET /api/v2/auth/login?next=https%3A%2F%2Fintegration-airflow-new.mht%2F HTTP/1.1" 307 Temporary Redirect
Oct 28 16:39:29 integration-airflow-new.mht python3.13[859127]: INFO:     127.0.0.1:39752 - "GET /ui/config HTTP/1.1" 401 Unauthorized
Oct 28 16:39:29 integration-airflow-new.mht python3.13[859127]: INFO:     127.0.0.1:39768 - "GET /ui/auth/menus HTTP/1.1" 401 Unauthorized
Oct 28 16:39:29 integration-airflow-new.mht python3.13[859125]: INFO:     127.0.0.1:39782 - "GET /api/v2/plugins HTTP/1.1" 401 Unauthorized
Oct 28 16:39:29 integration-airflow-new.mht python3.13[859124]: INFO:     127.0.0.1:39780 - "GET /api/v2/version HTTP/1.1" 200 OK
Oct 28 16:39:29 integration-airflow-new.mht python3.13[859127]: INFO:     127.0.0.1:39790 - "GET /static/i18n/locales/en/components.json HTTP/1.1" 200 OK
Oct 28 16:39:29 integration-airflow-new.mht python3.13[859126]: INFO:     127.0.0.1:39848 - "GET /api/v2/auth/login?next=https%3A%2F%2Fintegration-airflow-new.mht%2F HTTP/1.1" 307 Temporary Redirect
Oct 28 16:39:29 integration-airflow-new.mht python3.13[859125]: INFO:     127.0.0.1:39854 - "GET /auth/login/?next=https://integration-airflow-new.mht/ HTTP/1.1" 302 Found
Oct 28 16:39:30 integration-airflow-new.mht python3.13[859124]: INFO:     127.0.0.1:39856 - "GET /auth/ HTTP/1.1" 302 Found
Oct 28 16:39:30 integration-airflow-new.mht python3.13[859127]: INFO:     127.0.0.1:39868 - "GET /static/i18n/locales/en/components.json HTTP/1.1" 200 OK
Oct 28 16:39:30 integration-airflow-new.mht python3.13[859126]: INFO:     127.0.0.1:39874 - "GET /static/i18n/locales/en/browse.json HTTP/1.1" 200 OK
Oct 28 16:39:30 integration-airflow-new.mht python3.13[859127]: INFO:     127.0.0.1:39886 - "GET /static/i18n/locales/en/common.json HTTP/1.1" 200 OK
Oct 28 16:39:30 integration-airflow-new.mht python3.13[859127]: INFO:     127.0.0.1:39908 - "GET /static/i18n/locales/en/dags.json HTTP/1.1" 200 OK
Oct 28 16:39:30 integration-airflow-new.mht python3.13[859127]: INFO:     127.0.0.1:39914 - "GET /static/i18n/locales/en/admin.json HTTP/1.1" 200 OK
Oct 28 16:39:30 integration-airflow-new.mht python3.13[859124]: INFO:     127.0.0.1:39894 - "GET / HTTP/1.1" 200 OK
Oct 28 16:39:30 integration-airflow-new.mht python3.13[859125]: INFO:     127.0.0.1:39930 - "GET /static/i18n/locales/en/assets.json HTTP/1.1" 200 OK
Oct 28 16:39:30 integration-airflow-new.mht python3.13[859125]: INFO:     127.0.0.1:39938 - "GET /static/i18n/locales/en/dashboard.json HTTP/1.1" 200 OK
Oct 28 16:39:30 integration-airflow-new.mht python3.13[859125]: INFO:     127.0.0.1:39948 - "GET /ui/auth/menus HTTP/1.1" 200 OK
Oct 28 16:39:30 integration-airflow-new.mht python3.13[859125]: INFO:     127.0.0.1:56910 - "GET /api/v2/version HTTP/1.1" 200 OK
Oct 28 16:39:30 integration-airflow-new.mht python3.13[859125]: INFO:     127.0.0.1:56918 - "GET /static/i18n/locales/en/hitl.json HTTP/1.1" 200 OK
Oct 28 16:39:31 integration-airflow-new.mht python3.13[859124]: /opt/airflow/.local/lib/python3.13/site-packages/airflow/api_fastapi/core_api/routes/ui/config.py:52 DeprecationWarning: The hide_paused_dags_by_default option in [webserver] has been moved to the hide_paused_dags_by_default option in [api] - the old setting has been used, but please update your config.
Oct 28 16:39:31 integration-airflow-new.mht python3.13[859124]: /opt/airflow/.local/lib/python3.13/site-packages/airflow/api_fastapi/core_api/routes/ui/config.py:52 DeprecationWarning: The page_size option in [webserver] has been moved to the page_size option in [api] - the old setting has been used, but please update your config.
Oct 28 16:39:31 integration-airflow-new.mht python3.13[859124]: /opt/airflow/.local/lib/python3.13/site-packages/airflow/api_fastapi/core_api/routes/ui/config.py:52 DeprecationWarning: The default_wrap option in [webserver] has been moved to the default_wrap option in [api] - the old setting has been used, but please update your config.
Oct 28 16:39:31 integration-airflow-new.mht python3.13[859124]: /opt/airflow/.local/lib/python3.13/site-packages/airflow/api_fastapi/core_api/routes/ui/config.py:52 DeprecationWarning: The auto_refresh_interval option in [webserver] has been moved to the auto_refresh_interval option in [api] - the old setting has been used, but please update your config.
Oct 28 16:39:31 integration-airflow-new.mht python3.13[859124]: INFO:     127.0.0.1:56926 - "GET /ui/config HTTP/1.1" 200 OK
Oct 28 16:39:31 integration-airflow-new.mht python3.13[859127]: INFO:     127.0.0.1:56952 - "GET /static/i18n/locales/en/dag.json HTTP/1.1" 200 OK
Oct 28 16:39:31 integration-airflow-new.mht python3.13[859124]: INFO:     127.0.0.1:56932 - "GET /ui/auth/menus HTTP/1.1" 200 OK
Oct 28 16:39:31 integration-airflow-new.mht python3.13[859124]: INFO:     127.0.0.1:56942 - "GET /api/v2/version HTTP/1.1" 200 OK
Oct 28 16:39:31 integration-airflow-new.mht python3.13[859125]: INFO:     127.0.0.1:56974 - "GET /api/v2/plugins/importErrors HTTP/1.1" 200 OK
Oct 28 16:39:31 integration-airflow-new.mht python3.13[859127]: INFO:     127.0.0.1:56962 - "GET /api/v2/importErrors HTTP/1.1" 200 OK
Oct 28 16:39:31 integration-airflow-new.mht python3.13[859127]: INFO:     127.0.0.1:56966 - "GET /ui/dashboard/dag_stats HTTP/1.1" 200 OK
Oct 28 16:39:31 integration-airflow-new.mht python3.13[859127]: INFO:     127.0.0.1:56982 - "GET /ui/dashboard/historical_metrics_data?start_date=2025-10-27T16%3A39%3A31.068Z HTTP/1.1" 200 OK
Oct 28 16:39:31 integration-airflow-new.mht python3.13[859125]: INFO:     127.0.0.1:56996 - "GET /api/v2/assets/events?limit=6&order_by=-timestamp&timestamp_gte=2025-10-27T16%3A39%3A31.068Z&timestamp_lte=2025-10-28T16%3A39%3A31.068Z HTTP/1.1" 200 OK
Oct 28 16:39:31 integration-airflow-new.mht python3.13[859125]: INFO:     127.0.0.1:57020 - "GET /api/v2/plugins HTTP/1.1" 200 OK
Oct 28 16:39:31 integration-airflow-new.mht python3.13[859127]: INFO:     127.0.0.1:57036 - "GET /api/v2/dags/~/dagRuns?state=running&state=queued HTTP/1.1" 200 OK
Oct 28 16:39:31 integration-airflow-new.mht python3.13[859125]: INFO:     127.0.0.1:57058 - "GET /api/v2/importErrors?limit=15&offset=0 HTTP/1.1" 200 OK
Oct 28 16:39:31 integration-airflow-new.mht python3.13[859127]: INFO:     127.0.0.1:57052 - "GET /api/v2/monitor/health HTTP/1.1" 200 OK
Oct 28 16:39:31 integration-airflow-new.mht python3.13[859127]: INFO:     127.0.0.1:57004 - "GET /ui/dags?limit=10&is_favorite=true HTTP/1.1" 200 OK
Oct 28 16:39:31 integration-airflow-new.mht python3.13[859124]: INFO:     127.0.0.1:57048 - "GET /api/v2/dags/~/dagRuns/~/hitlDetails?state=deferred&response_received=false HTTP/1.1" 200 OK
Oct 28 16:39:31 integration-airflow-new.mht python3.13[859124]: INFO:     127.0.0.1:57054 - "GET /api/v2/pools HTTP/1.1" 200 OK
Oct 28 16:40:01 integration-airflow-new.mht python3.13[859125]: INFO:     127.0.0.1:56330 - "GET /api/v2/dags/~/dagRuns?state=running&state=queued HTTP/1.1" 200 OK
Oct 28 16:40:19 integration-airflow-new.mht python3.13[859127]: INFO:     127.0.0.1:38542 - "GET /api/v2/dagTags?limit=10&offset=0&order_by=name&tag_name_pattern= HTTP/1.1" 200 OK
Oct 28 16:40:19 integration-airflow-new.mht python3.13[859125]: INFO:     127.0.0.1:38552 - "GET /ui/dags?dag_runs_limit=14&limit=100&offset=0&exclude_stale=true&order_by=dag_display_name HTTP/1.1" 200 OK
Oct 28 16:40:20 integration-airflow-new.mht python3.13[859126]: INFO:     127.0.0.1:38578 - "GET /ui/next_run_assets/asset_alias_example_alias_consumer_with_no_taskflow HTTP/1.1" 200 OK
Oct 28 16:40:20 integration-airflow-new.mht python3.13[859124]: INFO:     127.0.0.1:38596 - "GET /ui/next_run_assets/asset_consumes_1_and_2 HTTP/1.1" 200 OK
Oct 28 16:40:20 integration-airflow-new.mht python3.13[859127]: INFO:     127.0.0.1:38586 - "GET /ui/next_run_assets/asset_consumes_1 HTTP/1.1" 200 OK
Oct 28 16:40:20 integration-airflow-new.mht python3.13[859127]: INFO:     127.0.0.1:38566 - "GET /ui/next_run_assets/asset_alias_example_alias_consumer HTTP/1.1" 200 OK
Oct 28 16:40:20 integration-airflow-new.mht python3.13[859127]: INFO:     127.0.0.1:38600 - "GET /ui/next_run_assets/asset_consumes_1_never_scheduled HTTP/1.1" 200 OK
Oct 28 16:40:20 integration-airflow-new.mht python3.13[859127]: INFO:     127.0.0.1:38604 - "GET /ui/next_run_assets/asset_consumes_unknown_never_scheduled HTTP/1.1" 200 OK
Oct 28 16:40:20 integration-airflow-new.mht python3.13[859127]: INFO:     127.0.0.1:38606 - "GET /ui/next_run_assets/asset_s3_bucket_consumer HTTP/1.1" 200 OK
Oct 28 16:40:20 integration-airflow-new.mht python3.13[859125]: INFO:     127.0.0.1:38616 - "GET /ui/next_run_assets/asset_s3_bucket_consumer_with_no_taskflow HTTP/1.1" 200 OK
Oct 28 16:40:20 integration-airflow-new.mht python3.13[859124]: INFO:     127.0.0.1:37124 - "GET /ui/next_run_assets/conditional_asset_and_time_based_timetable HTTP/1.1" 200 OK
Oct 28 16:40:20 integration-airflow-new.mht python3.13[859127]: INFO:     127.0.0.1:37138 - "GET /ui/next_run_assets/consume_1_or_2_with_asset_expressions HTTP/1.1" 200 OK
Oct 28 16:40:20 integration-airflow-new.mht python3.13[859125]: INFO:     127.0.0.1:37130 - "GET /ui/next_run_assets/consume_1_and_2_with_asset_expressions HTTP/1.1" 200 OK
Oct 28 16:40:20 integration-airflow-new.mht python3.13[859125]: INFO:     127.0.0.1:37142 - "GET /ui/next_run_assets/consume_1_or_both_2_and_3_with_asset_expressions HTTP/1.1" 200 OK
Oct 28 16:40:20 integration-airflow-new.mht python3.13[859127]: INFO:     127.0.0.1:37154 - "GET /ui/next_run_assets/consumes_asset_decorator HTTP/1.1" 200 OK
Oct 28 16:40:20 integration-airflow-new.mht python3.13[859127]: INFO:     127.0.0.1:37170 - "GET /ui/next_run_assets/example_asset_with_watchers HTTP/1.1" 200 OK
Oct 28 16:40:22 integration-airflow-new.mht python3.13[859127]: INFO:     127.0.0.1:37184 - "GET /ui/dashboard/historical_metrics_data?start_date=2025-10-27T16%3A40%3A21.846Z HTTP/1.1" 200 OK
Oct 28 16:40:22 integration-airflow-new.mht python3.13[859127]: INFO:     127.0.0.1:37174 - "GET /api/v2/assets/events?limit=6&order_by=-timestamp&timestamp_gte=2025-10-27T16%3A40%3A21.846Z&timestamp_lte=2025-10-28T16%3A40%3A21.846Z HTTP/1.1" 200 OK
Oct 28 16:40:27 integration-airflow-new.mht python3.13[859127]: /opt/airflow/.local/lib/python3.13/site-packages/airflow/api_fastapi/core_api/services/public/config.py:29 DeprecationWarning: The expose_config option in [webserver] has been moved to the expose_config option in [api] - the old setting has been used, but please update your config.
Oct 28 16:40:27 integration-airflow-new.mht python3.13[859127]: /opt/airflow/.local/lib/python3.13/site-packages/airflow/api_fastapi/core_api/services/public/config.py:33 DeprecationWarning: The expose_config option in [webserver] has been moved to the expose_config option in [api] - the old setting has been used, but please update your config.
Oct 28 16:40:27 integration-airflow-new.mht python3.13[859127]: INFO:     127.0.0.1:37188 - "GET /api/v2/config HTTP/1.1" 403 Forbidden
Oct 28 16:40:33 integration-airflow-new.mht python3.13[859127]: INFO:     127.0.0.1:38918 - "GET /api/v2/pools?limit=100&offset=0&order_by=name HTTP/1.1" 200 OK
Oct 28 16:40:52 integration-airflow-new.mht python3.13[859127]: INFO:     127.0.0.1:59174 - "GET /api/v2/assets/events?limit=6&order_by=-timestamp&timestamp_gte=2025-10-27T16%3A40%3A51.859Z&timestamp_lte=2025-10-28T16%3A40%3A51.859Z HTTP/1.1" 200 OK
Oct 28 16:40:52 integration-airflow-new.mht python3.13[859125]: INFO:     127.0.0.1:59186 - "GET /ui/dashboard/historical_metrics_data?start_date=2025-10-27T16%3A40%3A51.859Z HTTP/1.1" 200 OK
Oct 28 16:41:22 integration-airflow-new.mht python3.13[859127]: INFO:     127.0.0.1:33724 - "GET /api/v2/dags/~/dagRuns?state=running&state=queued HTTP/1.1" 200 OK
Oct 28 16:41:52 integration-airflow-new.mht python3.13[859125]: INFO:     127.0.0.1:35568 - "GET /api/v2/dags/~/dagRuns?state=running&state=queued HTTP/1.1" 200 OK
Oct 28 16:42:22 integration-airflow-new.mht python3.13[859125]: INFO:     127.0.0.1:59920 - "GET /api/v2/dags/~/dagRuns?state=running&state=queued HTTP/1.1" 200 OK
Oct 28 16:42:52 integration-airflow-new.mht python3.13[859125]: INFO:     127.0.0.1:38480 - "GET /api/v2/dags/~/dagRuns?state=running&state=queued HTTP/1.1" 200 OK
Oct 28 16:43:22 integration-airflow-new.mht python3.13[859127]: INFO:     127.0.0.1:46440 - "GET /api/v2/assets?limit=100&offset=0&name_pattern= HTTP/1.1" 200 OK
Oct 28 16:52:21 integration-airflow-new.mht python3.13[859127]: INFO:     127.0.0.1:45728 - "GET /ui/dags?dag_runs_limit=1&limit=100&offset=0&exclude_stale=true&paused=false&order_by=dag_display_name HTTP/1.1" 200 OK
Oct 28 16:52:22 integration-airflow-new.mht python3.13[859127]: /opt/airflow/.local/lib/python3.13/site-packages/airflow/api_fastapi/core_api/routes/ui/config.py:52 DeprecationWarning: The hide_paused_dags_by_default option in [webserver] has been moved to the hide_paused_dags_by_default option in [api] - the old setting has been used, but please update your config.
Oct 28 16:52:22 integration-airflow-new.mht python3.13[859127]: /opt/airflow/.local/lib/python3.13/site-packages/airflow/api_fastapi/core_api/routes/ui/config.py:52 DeprecationWarning: The page_size option in [webserver] has been moved to the page_size option in [api] - the old setting has been used, but please update your config.
Oct 28 16:52:22 integration-airflow-new.mht python3.13[859127]: /opt/airflow/.local/lib/python3.13/site-packages/airflow/api_fastapi/core_api/routes/ui/config.py:52 DeprecationWarning: The default_wrap option in [webserver] has been moved to the default_wrap option in [api] - the old setting has been used, but please update your config.
Oct 28 16:52:22 integration-airflow-new.mht python3.13[859127]: /opt/airflow/.local/lib/python3.13/site-packages/airflow/api_fastapi/core_api/routes/ui/config.py:52 DeprecationWarning: The auto_refresh_interval option in [webserver] has been moved to the auto_refresh_interval option in [api] - the old setting has been used, but please update your config.
Oct 28 16:52:22 integration-airflow-new.mht python3.13[859127]: INFO:     127.0.0.1:45742 - "GET /ui/config HTTP/1.1" 200 OK
Oct 28 16:52:22 integration-airflow-new.mht python3.13[859127]: INFO:     127.0.0.1:45746 - "GET /ui/dags?is_favorite=true HTTP/1.1" 200 OK
Oct 28 16:55:31 integration-airflow-new.mht python3.13[859127]: INFO:     127.0.0.1:45264 - "GET /ui/config HTTP/1.1" 200 OK
Oct 28 16:55:31 integration-airflow-new.mht python3.13[859126]: INFO:     127.0.0.1:45306 - "GET /api/v2/plugins/importErrors HTTP/1.1" 200 OK
Oct 28 16:55:31 integration-airflow-new.mht python3.13[859127]: INFO:     127.0.0.1:45268 - "GET /api/v2/dags/~/dagRuns?state=running&state=queued HTTP/1.1" 200 OK
Oct 28 16:55:31 integration-airflow-new.mht python3.13[859125]: INFO:     127.0.0.1:45328 - "GET /api/v2/assets/events?limit=6&order_by=-timestamp&timestamp_gte=2025-10-27T16%3A55%3A31.212Z&timestamp_lte=2025-10-28T16%3A55%3A31.212Z HTTP/1.1" 200 OK
Oct 28 16:55:31 integration-airflow-new.mht python3.13[859125]: INFO:     127.0.0.1:45292 - "GET /api/v2/importErrors HTTP/1.1" 200 OK
Oct 28 16:55:31 integration-airflow-new.mht python3.13[859126]: INFO:     127.0.0.1:45296 - "GET /ui/dashboard/historical_metrics_data?start_date=2025-10-27T16%3A55%3A31.212Z HTTP/1.1" 200 OK
Oct 28 16:55:31 integration-airflow-new.mht python3.13[859124]: INFO:     127.0.0.1:45270 - "GET /ui/dashboard/dag_stats HTTP/1.1" 200 OK
Oct 28 16:55:31 integration-airflow-new.mht python3.13[859127]: INFO:     127.0.0.1:45342 - "GET /api/v2/monitor/health HTTP/1.1" 200 OK
Oct 28 16:55:31 integration-airflow-new.mht python3.13[859124]: INFO:     127.0.0.1:45320 - "GET /api/v2/dags/~/dagRuns/~/hitlDetails?state=deferred&response_received=false HTTP/1.1" 200 OK
Oct 28 16:55:31 integration-airflow-new.mht python3.13[859127]: INFO:     127.0.0.1:45354 - "GET /api/v2/plugins HTTP/1.1" 200 OK
Oct 28 16:55:31 integration-airflow-new.mht python3.13[859125]: INFO:     127.0.0.1:45366 - "GET /ui/auth/menus HTTP/1.1" 200 OK
Oct 28 16:55:31 integration-airflow-new.mht python3.13[859127]: INFO:     127.0.0.1:45382 - "GET /api/v2/importErrors?limit=15&offset=0 HTTP/1.1" 200 OK
Oct 28 16:55:31 integration-airflow-new.mht python3.13[859124]: INFO:     127.0.0.1:45312 - "GET /ui/dags?limit=10&is_favorite=true HTTP/1.1" 200 OK
Oct 28 16:55:31 integration-airflow-new.mht python3.13[859124]: INFO:     127.0.0.1:45380 - "GET /api/v2/pools HTTP/1.1" 200 OK
Oct 28 16:56:01 integration-airflow-new.mht python3.13[859125]: INFO:     127.0.0.1:33916 - "GET /api/v2/dags/~/dagRuns?state=running&state=queued HTTP/1.1" 200 OK
Oct 28 16:56:32 integration-airflow-new.mht python3.13[859127]: INFO:     127.0.0.1:45832 - "GET /api/v2/dags/~/dagRuns?state=running&state=queued HTTP/1.1" 200 OK
Oct 28 16:57:02 integration-airflow-new.mht python3.13[859125]: INFO:     127.0.0.1:49524 - "GET /api/v2/dags/~/dagRuns?state=running&state=queued HTTP/1.1" 200 OK
Oct 28 16:57:32 integration-airflow-new.mht python3.13[859127]: INFO:     127.0.0.1:50834 - "GET /api/v2/dags/~/dagRuns?state=running&state=queued HTTP/1.1" 200 OK
Oct 28 16:58:02 integration-airflow-new.mht python3.13[859127]: INFO:     127.0.0.1:59562 - "GET /api/v2/dags/~/dagRuns?state=running&state=queued HTTP/1.1" 200 OK
Oct 28 20:30:06 integration-airflow-new.mht python3.13[859124]: INFO:     127.0.0.1:46536 - "GET /api/v2/plugins HTTP/1.1" 200 OK
Oct 28 20:30:06 integration-airflow-new.mht python3.13[859125]: /opt/airflow/.local/lib/python3.13/site-packages/airflow/api_fastapi/core_api/routes/ui/config.py:52 DeprecationWarning: The hide_paused_dags_by_default option in [webserver] has been moved to the hide_paused_dags_by_default option in [api] - the old setting has been used, but please update your config.
Oct 28 20:30:06 integration-airflow-new.mht python3.13[859125]: /opt/airflow/.local/lib/python3.13/site-packages/airflow/api_fastapi/core_api/routes/ui/config.py:52 DeprecationWarning: The page_size option in [webserver] has been moved to the page_size option in [api] - the old setting has been used, but please update your config.
Oct 28 20:30:06 integration-airflow-new.mht python3.13[859125]: /opt/airflow/.local/lib/python3.13/site-packages/airflow/api_fastapi/core_api/routes/ui/config.py:52 DeprecationWarning: The default_wrap option in [webserver] has been moved to the default_wrap option in [api] - the old setting has been used, but please update your config.
Oct 28 20:30:06 integration-airflow-new.mht python3.13[859125]: /opt/airflow/.local/lib/python3.13/site-packages/airflow/api_fastapi/core_api/routes/ui/config.py:52 DeprecationWarning: The auto_refresh_interval option in [webserver] has been moved to the auto_refresh_interval option in [api] - the old setting has been used, but please update your config.
Oct 28 20:30:06 integration-airflow-new.mht python3.13[859125]: INFO:     127.0.0.1:46526 - "GET /ui/config HTTP/1.1" 200 OK
Oct 28 20:30:06 integration-airflow-new.mht python3.13[859127]: INFO:     127.0.0.1:46552 - "GET /ui/auth/menus HTTP/1.1" 200 OK
Oct 28 20:30:06 integration-airflow-new.mht python3.13[859125]: INFO:     127.0.0.1:46566 - "GET /api/v2/dags/~/dagRuns?state=running&state=queued HTTP/1.1" 200 OK
Oct 28 20:30:06 integration-airflow-new.mht python3.13[859125]: INFO:     127.0.0.1:46584 - "GET /api/v2/version HTTP/1.1" 200 OK
Oct 28 20:30:06 integration-airflow-new.mht python3.13[859125]: INFO:     127.0.0.1:46600 - "GET /ui/dashboard/dag_stats HTTP/1.1" 200 OK
Oct 28 20:30:06 integration-airflow-new.mht python3.13[859127]: INFO:     127.0.0.1:46576 - "GET /api/v2/dags/~/dagRuns/~/hitlDetails?state=deferred&response_received=false HTTP/1.1" 200 OK
Oct 28 20:30:06 integration-airflow-new.mht python3.13[859127]: INFO:     127.0.0.1:46614 - "GET /api/v2/importErrors HTTP/1.1" 200 OK
Oct 28 20:30:06 integration-airflow-new.mht python3.13[859127]: INFO:     127.0.0.1:46618 - "GET /api/v2/plugins/importErrors HTTP/1.1" 200 OK
Oct 28 20:30:06 integration-airflow-new.mht python3.13[859127]: INFO:     127.0.0.1:46630 - "GET /ui/dags?limit=10&is_favorite=true HTTP/1.1" 200 OK
Oct 28 20:30:06 integration-airflow-new.mht python3.13[859124]: INFO:     127.0.0.1:46644 - "GET /api/v2/monitor/health HTTP/1.1" 200 OK
Oct 28 20:30:06 integration-airflow-new.mht python3.13[859127]: INFO:     127.0.0.1:46660 - "GET /api/v2/pools HTTP/1.1" 200 OK
Oct 28 20:30:06 integration-airflow-new.mht python3.13[859127]: INFO:     127.0.0.1:46670 - "GET /ui/dashboard/historical_metrics_data?start_date=2025-10-27T16%3A55%3A31.212Z HTTP/1.1" 200 OK
Oct 28 20:30:06 integration-airflow-new.mht python3.13[859127]: INFO:     127.0.0.1:46684 - "GET /api/v2/assets/events?limit=6&order_by=-timestamp&timestamp_gte=2025-10-27T16%3A55%3A31.212Z&timestamp_lte=2025-10-28T16%3A55%3A31.212Z HTTP/1.1" 200 OK
Oct 28 20:30:06 integration-airflow-new.mht python3.13[859127]: INFO:     127.0.0.1:46686 - "GET /api/v2/importErrors?limit=15&offset=0 HTTP/1.1" 200 OK
Oct 29 15:00:26 integration-airflow-new.mht python3.13[859125]: INFO:     127.0.0.1:51206 - "GET /dags?paused=false HTTP/1.1" 200 OK
Oct 29 15:00:27 integration-airflow-new.mht python3.13[859127]: INFO:     127.0.0.1:51216 - "GET /static/assets/index-BjF9k7EU.js HTTP/1.1" 200 OK
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859127]: INFO:     127.0.0.1:51222 - "GET /ui/config HTTP/1.1" 401 Unauthorized
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859127]: INFO:     127.0.0.1:51232 - "GET /static/i18n/locales/en/common.json HTTP/1.1" 200 OK
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859127]: INFO:     127.0.0.1:51236 - "GET /api/v2/auth/login?next=https%3A%2F%2F172.23.4.38%2Fdags%3Fpaused%3Dfalse HTTP/1.1" 307 Temporary Redirect
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859127]: INFO:     127.0.0.1:51246 - "GET /static/i18n/locales/en/components.json HTTP/1.1" 200 OK
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]: [2025-10-29T15:00:29.482378Z] {app.py:1744} ERROR - Exception on /login/ [GET]
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]: Traceback (most recent call last):
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   File "/opt/airflow/.local/lib/python3.13/site-packages/sqlalchemy/engine/base.py", line 1967, in _exec_single_context
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     self.dialect.do_execute(
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     ~~~~~~~~~~~~~~~~~~~~~~~^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:         cursor, str_statement, effective_parameters, context
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     )
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     ^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   File "/opt/airflow/.local/lib/python3.13/site-packages/sqlalchemy/engine/default.py", line 951, in do_execute
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     cursor.execute(statement, parameters)
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   File "/opt/airflow/.local/lib/python3.13/site-packages/MySQLdb/cursors.py", line 179, in execute
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     res = self._query(mogrified_query)
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   File "/opt/airflow/.local/lib/python3.13/site-packages/MySQLdb/cursors.py", line 330, in _query
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     db.query(q)
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     ~~~~~~~~^^^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   File "/opt/airflow/.local/lib/python3.13/site-packages/MySQLdb/connections.py", line 280, in query
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     _mysql.connection.query(self, query)
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]: MySQLdb.OperationalError: (4031, 'The client was disconnected by the server because of inactivity. See wait_timeout and interactive_timeout for configuring this behavior.')
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]: The above exception was the direct cause of the following exception:
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]: Traceback (most recent call last):
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   File "/opt/airflow/.local/lib/python3.13/site-packages/flask/app.py", line 2529, in wsgi_app
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     response = self.full_dispatch_request()
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   File "/opt/airflow/.local/lib/python3.13/site-packages/flask/app.py", line 1825, in full_dispatch_request
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     rv = self.handle_user_exception(e)
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   File "/opt/airflow/.local/lib/python3.13/site-packages/flask/app.py", line 1821, in full_dispatch_request
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     rv = self.preprocess_request()
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   File "/opt/airflow/.local/lib/python3.13/site-packages/flask/app.py", line 2313, in preprocess_request
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     rv = self.ensure_sync(before_func)()
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   File "/opt/airflow/.local/lib/python3.13/site-packages/airflow/providers/fab/www/security_manager.py", line 57, in before_request
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     g.user = get_auth_manager().get_user()
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:              ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   File "/opt/airflow/.local/lib/python3.13/site-packages/airflow/providers/fab/auth_manager/fab_auth_manager.py", line 278, in get_user
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     if current_user.is_anonymous and getattr(g, "user", None) is not None and not g.user.is_anonymous:
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:        ^^^^^^^^^^^^^^^^^^^^^^^^^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   File "/opt/airflow/.local/lib/python3.13/site-packages/werkzeug/local.py", line 316, in __get__
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     obj = instance._get_current_object()
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   File "/opt/airflow/.local/lib/python3.13/site-packages/werkzeug/local.py", line 520, in _get_current_object
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     return get_name(local())  # type: ignore
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:                     ~~~~~^^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   File "/opt/airflow/.local/lib/python3.13/site-packages/flask_login/utils.py", line 25, in <lambda>
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     current_user = LocalProxy(lambda: _get_user())
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:                                       ~~~~~~~~~^^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   File "/opt/airflow/.local/lib/python3.13/site-packages/flask_login/utils.py", line 370, in _get_user
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     current_app.login_manager._load_user()
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   File "/opt/airflow/.local/lib/python3.13/site-packages/flask_login/login_manager.py", line 364, in _load_user
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     user = self._user_callback(user_id)
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   File "/opt/airflow/.local/lib/python3.13/site-packages/airflow/providers/fab/auth_manager/security_manager/override.py", line 1384, in load_user
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     user = self.get_user_by_id(int(pk))
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   File "/opt/airflow/.local/lib/python3.13/site-packages/airflow/providers/fab/auth_manager/security_manager/override.py", line 1390, in get_user_by_id
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     return self.session.get(self.user_model, pk)
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:            ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   File "/opt/airflow/.local/lib/python3.13/site-packages/sqlalchemy/orm/session.py", line 3680, in get
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     return self._get_impl(
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:            ~~~~~~~~~~~~~~^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:         entity,
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:         ^^^^^^^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     ...<7 lines>...
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:         bind_arguments=bind_arguments,
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     )
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     ^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   File "/opt/airflow/.local/lib/python3.13/site-packages/sqlalchemy/orm/session.py", line 3859, in _get_impl
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     return db_load_fn(
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:         self,
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     ...<5 lines>...
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:         bind_arguments=bind_arguments,
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     )
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   File "/opt/airflow/.local/lib/python3.13/site-packages/sqlalchemy/orm/loading.py", line 695, in load_on_pk_identity
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     session.execute(
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     ~~~~~~~~~~~~~~~^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:         q,
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:         ^^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     ...<2 lines>...
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:         bind_arguments=bind_arguments,
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     )
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     ^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   File "/opt/airflow/.local/lib/python3.13/site-packages/sqlalchemy/orm/session.py", line 2351, in execute
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     return self._execute_internal(
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:            ~~~~~~~~~~~~~~~~~~~~~~^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:         statement,
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:         ^^^^^^^^^^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     ...<4 lines>...
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:         _add_event=_add_event,
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:         ^^^^^^^^^^^^^^^^^^^^^^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     )
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     ^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   File "/opt/airflow/.local/lib/python3.13/site-packages/sqlalchemy/orm/session.py", line 2249, in _execute_internal
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     result: Result[Any] = compile_state_cls.orm_execute_statement(
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:                           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:         self,
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:         ^^^^^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     ...<4 lines>...
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:         conn,
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:         ^^^^^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     )
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     ^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   File "/opt/airflow/.local/lib/python3.13/site-packages/sqlalchemy/orm/context.py", line 306, in orm_execute_statement
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     result = conn.execute(
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:         statement, params or {}, execution_options=execution_options
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     )
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   File "/opt/airflow/.local/lib/python3.13/site-packages/sqlalchemy/engine/base.py", line 1419, in execute
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     return meth(
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:         self,
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:         distilled_parameters,
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:         execution_options or NO_OPTIONS,
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     )
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   File "/opt/airflow/.local/lib/python3.13/site-packages/sqlalchemy/sql/elements.py", line 526, in _execute_on_connection
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     return connection._execute_clauseelement(
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:         self, distilled_params, execution_options
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     )
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     ^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   File "/opt/airflow/.local/lib/python3.13/site-packages/sqlalchemy/engine/base.py", line 1641, in _execute_clauseelement
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     ret = self._execute_context(
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:         dialect,
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     ...<8 lines>...
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:         cache_hit=cache_hit,
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     )
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   File "/opt/airflow/.local/lib/python3.13/site-packages/sqlalchemy/engine/base.py", line 1846, in _execute_context
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     return self._exec_single_context(
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:            ~~~~~~~~~~~~~~~~~~~~~~~~~^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:         dialect, context, statement, parameters
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     )
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     ^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   File "/opt/airflow/.local/lib/python3.13/site-packages/sqlalchemy/engine/base.py", line 1986, in _exec_single_context
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     self._handle_dbapi_exception(
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:         e, str_statement, effective_parameters, cursor, context
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     )
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     ^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   File "/opt/airflow/.local/lib/python3.13/site-packages/sqlalchemy/engine/base.py", line 2355, in _handle_dbapi_exception
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   File "/opt/airflow/.local/lib/python3.13/site-packages/sqlalchemy/engine/base.py", line 1967, in _exec_single_context
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     self.dialect.do_execute(
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     ~~~~~~~~~~~~~~~~~~~~~~~^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:         cursor, str_statement, effective_parameters, context
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     )
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     ^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   File "/opt/airflow/.local/lib/python3.13/site-packages/sqlalchemy/engine/default.py", line 951, in do_execute
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     cursor.execute(statement, parameters)
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   File "/opt/airflow/.local/lib/python3.13/site-packages/MySQLdb/cursors.py", line 179, in execute
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     res = self._query(mogrified_query)
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   File "/opt/airflow/.local/lib/python3.13/site-packages/MySQLdb/cursors.py", line 330, in _query
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     db.query(q)
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     ~~~~~~~~^^^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   File "/opt/airflow/.local/lib/python3.13/site-packages/MySQLdb/connections.py", line 280, in query
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     _mysql.connection.query(self, query)
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]: sqlalchemy.exc.OperationalError: (MySQLdb.OperationalError) (4031, 'The client was disconnected by the server because of inactivity. See wait_timeout and interactive_timeout for configuring this behavior.')
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]: [SQL: SELECT ab_user.id AS ab_user_id, ab_user.first_name AS ab_user_first_name, ab_user.last_name AS ab_user_last_name, ab_user.username AS ab_user_username, ab_user.password AS ab_user_password, ab_user.active AS ab_user_active, ab_user.email AS ab_user_email, ab_user.last_login AS ab_user_last_login, ab_user.login_count AS ab_user_login_count, ab_user.fail_login_count AS ab_user_fail_login_count, ab_user.created_on AS ab_user_created_on, ab_user.changed_on AS ab_user_changed_on, ab_user.created_by_fk AS ab_user_created_by_fk, ab_user.changed_by_fk AS ab_user_changed_by_fk
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]: FROM ab_user
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]: WHERE ab_user.id = %s]
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]: [parameters: (1,)]
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]: (Background on this error at: https://sqlalche.me/e/20/e3q8)
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]: INFO:     127.0.0.1:51254 - "GET /auth/login/?next=https://172.23.4.38/dags?paused=false HTTP/1.1" 500 Internal Server Error
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859124]: INFO:     127.0.0.1:51266 - "GET /static/i18n/locales/en/hitl.json HTTP/1.1" 200 OK
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859127]: INFO:     127.0.0.1:51278 - "GET /static/i18n/locales/en/dashboard.json HTTP/1.1" 200 OK
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859127]: INFO:     127.0.0.1:51290 - "GET /static/i18n/locales/en/admin.json HTTP/1.1" 200 OK
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]: ERROR:    Exception in ASGI application
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   + Exception Group Traceback (most recent call last):
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   |   File "/opt/airflow/.local/lib/python3.13/site-packages/uvicorn/protocols/http/httptools_impl.py", line 409, in run_asgi
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   |     result = await app(  # type: ignore[func-returns-value]
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   |         self.scope, self.receive, self.send
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   |     )
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   |     ^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   |   File "/opt/airflow/.local/lib/python3.13/site-packages/fastapi/applications.py", line 1082, in __call__
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   |     await super().__call__(scope, receive, send)
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   |   File "/opt/airflow/.local/lib/python3.13/site-packages/starlette/applications.py", line 113, in __call__
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   |     await self.middleware_stack(scope, receive, send)
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   |   File "/opt/airflow/.local/lib/python3.13/site-packages/starlette/middleware/errors.py", line 186, in __call__
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   |     raise exc
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   |   File "/opt/airflow/.local/lib/python3.13/site-packages/starlette/middleware/errors.py", line 164, in __call__
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   |     await self.app(scope, receive, _send)
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   |   File "/opt/airflow/.local/lib/python3.13/site-packages/starlette/middleware/gzip.py", line 29, in __call__
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   |     await responder(scope, receive, send)
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   |   File "/opt/airflow/.local/lib/python3.13/site-packages/starlette/middleware/gzip.py", line 130, in __call__
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   |     await super().__call__(scope, receive, send)
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   |   File "/opt/airflow/.local/lib/python3.13/site-packages/starlette/middleware/gzip.py", line 46, in __call__
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   |     await self.app(scope, receive, self.send_with_compression)
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   |   File "/opt/airflow/.local/lib/python3.13/site-packages/starlette/middleware/cors.py", line 85, in __call__
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   |     await self.app(scope, receive, send)
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   |   File "/opt/airflow/.local/lib/python3.13/site-packages/starlette/middleware/exceptions.py", line 63, in __call__
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   |     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   |   File "/opt/airflow/.local/lib/python3.13/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   |     raise exc
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   |   File "/opt/airflow/.local/lib/python3.13/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   |     await app(scope, receive, sender)
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   |   File "/opt/airflow/.local/lib/python3.13/site-packages/starlette/routing.py", line 716, in __call__
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   |     await self.middleware_stack(scope, receive, send)
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   |   File "/opt/airflow/.local/lib/python3.13/site-packages/starlette/routing.py", line 736, in app
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   |     await route.handle(scope, receive, send)
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   |   File "/opt/airflow/.local/lib/python3.13/site-packages/starlette/routing.py", line 462, in handle
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   |     await self.app(scope, receive, send)
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   |   File "/opt/airflow/.local/lib/python3.13/site-packages/fastapi/applications.py", line 1082, in __call__
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   |     await super().__call__(scope, receive, send)
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   |   File "/opt/airflow/.local/lib/python3.13/site-packages/starlette/applications.py", line 113, in __call__
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   |     await self.middleware_stack(scope, receive, send)
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   |   File "/opt/airflow/.local/lib/python3.13/site-packages/starlette/middleware/errors.py", line 186, in __call__
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   |     raise exc
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   |   File "/opt/airflow/.local/lib/python3.13/site-packages/starlette/middleware/errors.py", line 164, in __call__
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   |     await self.app(scope, receive, _send)
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   |   File "/opt/airflow/.local/lib/python3.13/site-packages/starlette/middleware/exceptions.py", line 63, in __call__
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   |     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   |   File "/opt/airflow/.local/lib/python3.13/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   |     raise exc
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   |   File "/opt/airflow/.local/lib/python3.13/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   |     await app(scope, receive, sender)
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   |   File "/opt/airflow/.local/lib/python3.13/site-packages/starlette/routing.py", line 716, in __call__
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   |     await self.middleware_stack(scope, receive, send)
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   |   File "/opt/airflow/.local/lib/python3.13/site-packages/starlette/routing.py", line 736, in app
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   |     await route.handle(scope, receive, send)
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   |   File "/opt/airflow/.local/lib/python3.13/site-packages/starlette/routing.py", line 462, in handle
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   |     await self.app(scope, receive, send)
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   |   File "/opt/airflow/.local/lib/python3.13/site-packages/starlette/middleware/wsgi.py", line 81, in __call__
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   |     await responder(receive, send)
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   |   File "/opt/airflow/.local/lib/python3.13/site-packages/starlette/middleware/wsgi.py", line 106, in __call__
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   |     async with anyio.create_task_group() as task_group:
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   |                ~~~~~~~~~~~~~~~~~~~~~~~^^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   |   File "/opt/airflow/.local/lib/python3.13/site-packages/anyio/_backends/_asyncio.py", line 781, in __aexit__
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   |     raise BaseExceptionGroup(
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   |         "unhandled errors in a TaskGroup", self._exceptions
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   |     ) from None
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   | ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:   +-+---------------- 1 ----------------
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     | Traceback (most recent call last):
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |   File "/opt/airflow/.local/lib/python3.13/site-packages/sqlalchemy/engine/base.py", line 1967, in _exec_single_context
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     self.dialect.do_execute(
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     ~~~~~~~~~~~~~~~~~~~~~~~^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |         cursor, str_statement, effective_parameters, context
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     )
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     ^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |   File "/opt/airflow/.local/lib/python3.13/site-packages/sqlalchemy/engine/default.py", line 951, in do_execute
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     cursor.execute(statement, parameters)
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |   File "/opt/airflow/.local/lib/python3.13/site-packages/MySQLdb/cursors.py", line 179, in execute
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     res = self._query(mogrified_query)
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |   File "/opt/airflow/.local/lib/python3.13/site-packages/MySQLdb/cursors.py", line 330, in _query
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     db.query(q)
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     ~~~~~~~~^^^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |   File "/opt/airflow/.local/lib/python3.13/site-packages/MySQLdb/connections.py", line 280, in query
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     _mysql.connection.query(self, query)
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     | MySQLdb.OperationalError: (4031, 'The client was disconnected by the server because of inactivity. See wait_timeout and interactive_timeout for configuring this behavior.')
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     | The above exception was the direct cause of the following exception:
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     | Traceback (most recent call last):
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |   File "/opt/airflow/.local/lib/python3.13/site-packages/flask/app.py", line 2529, in wsgi_app
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     response = self.full_dispatch_request()
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |   File "/opt/airflow/.local/lib/python3.13/site-packages/flask/app.py", line 1825, in full_dispatch_request
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     rv = self.handle_user_exception(e)
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |   File "/opt/airflow/.local/lib/python3.13/site-packages/flask/app.py", line 1821, in full_dispatch_request
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     rv = self.preprocess_request()
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |   File "/opt/airflow/.local/lib/python3.13/site-packages/flask/app.py", line 2313, in preprocess_request
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     rv = self.ensure_sync(before_func)()
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |   File "/opt/airflow/.local/lib/python3.13/site-packages/airflow/providers/fab/www/security_manager.py", line 57, in before_request
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     g.user = get_auth_manager().get_user()
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |              ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |   File "/opt/airflow/.local/lib/python3.13/site-packages/airflow/providers/fab/auth_manager/fab_auth_manager.py", line 278, in get_user
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     if current_user.is_anonymous and getattr(g, "user", None) is not None and not g.user.is_anonymous:
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |        ^^^^^^^^^^^^^^^^^^^^^^^^^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |   File "/opt/airflow/.local/lib/python3.13/site-packages/werkzeug/local.py", line 316, in __get__
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     obj = instance._get_current_object()
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |   File "/opt/airflow/.local/lib/python3.13/site-packages/werkzeug/local.py", line 520, in _get_current_object
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     return get_name(local())  # type: ignore
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |                     ~~~~~^^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |   File "/opt/airflow/.local/lib/python3.13/site-packages/flask_login/utils.py", line 25, in <lambda>
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     current_user = LocalProxy(lambda: _get_user())
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |                                       ~~~~~~~~~^^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |   File "/opt/airflow/.local/lib/python3.13/site-packages/flask_login/utils.py", line 370, in _get_user
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     current_app.login_manager._load_user()
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |   File "/opt/airflow/.local/lib/python3.13/site-packages/flask_login/login_manager.py", line 364, in _load_user
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     user = self._user_callback(user_id)
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |   File "/opt/airflow/.local/lib/python3.13/site-packages/airflow/providers/fab/auth_manager/security_manager/override.py", line 1384, in load_user
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     user = self.get_user_by_id(int(pk))
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |   File "/opt/airflow/.local/lib/python3.13/site-packages/airflow/providers/fab/auth_manager/security_manager/override.py", line 1390, in get_user_by_id
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     return self.session.get(self.user_model, pk)
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |            ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |   File "/opt/airflow/.local/lib/python3.13/site-packages/sqlalchemy/orm/session.py", line 3680, in get
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     return self._get_impl(
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |            ~~~~~~~~~~~~~~^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |         entity,
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     db.query(q)
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     ~~~~~~~~^^^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |   File "/opt/airflow/.local/lib/python3.13/site-packages/MySQLdb/connections.py", line 280, in query
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     _mysql.connection.query(self, query)
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     | MySQLdb.OperationalError: (4031, 'The client was disconnected by the server because of inactivity. See wait_timeout and interactive_timeout for configuring this behavior.')
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     | The above exception was the direct cause of the following exception:
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     | Traceback (most recent call last):
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |   File "/opt/airflow/.local/lib/python3.13/site-packages/flask/app.py", line 2529, in wsgi_app
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     response = self.full_dispatch_request()
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |   File "/opt/airflow/.local/lib/python3.13/site-packages/flask/app.py", line 1825, in full_dispatch_request
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     rv = self.handle_user_exception(e)
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |   File "/opt/airflow/.local/lib/python3.13/site-packages/flask/app.py", line 1821, in full_dispatch_request
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     rv = self.preprocess_request()
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |   File "/opt/airflow/.local/lib/python3.13/site-packages/flask/app.py", line 2313, in preprocess_request
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     rv = self.ensure_sync(before_func)()
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |   File "/opt/airflow/.local/lib/python3.13/site-packages/airflow/providers/fab/www/security_manager.py", line 57, in before_request
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     g.user = get_auth_manager().get_user()
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |              ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |   File "/opt/airflow/.local/lib/python3.13/site-packages/airflow/providers/fab/auth_manager/fab_auth_manager.py", line 278, in get_user
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     if current_user.is_anonymous and getattr(g, "user", None) is not None and not g.user.is_anonymous:
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |        ^^^^^^^^^^^^^^^^^^^^^^^^^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |   File "/opt/airflow/.local/lib/python3.13/site-packages/werkzeug/local.py", line 316, in __get__
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     obj = instance._get_current_object()
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |   File "/opt/airflow/.local/lib/python3.13/site-packages/werkzeug/local.py", line 520, in _get_current_object
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     return get_name(local())  # type: ignore
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |                     ~~~~~^^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |   File "/opt/airflow/.local/lib/python3.13/site-packages/flask_login/utils.py", line 25, in <lambda>
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     current_user = LocalProxy(lambda: _get_user())
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |                                       ~~~~~~~~~^^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |   File "/opt/airflow/.local/lib/python3.13/site-packages/flask_login/utils.py", line 370, in _get_user
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     current_app.login_manager._load_user()
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |   File "/opt/airflow/.local/lib/python3.13/site-packages/flask_login/login_manager.py", line 364, in _load_user
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     user = self._user_callback(user_id)
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |   File "/opt/airflow/.local/lib/python3.13/site-packages/airflow/providers/fab/auth_manager/security_manager/override.py", line 1384, in load_user
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     user = self.get_user_by_id(int(pk))
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |   File "/opt/airflow/.local/lib/python3.13/site-packages/airflow/providers/fab/auth_manager/security_manager/override.py", line 1390, in get_user_by_id
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     return self.session.get(self.user_model, pk)
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |            ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |   File "/opt/airflow/.local/lib/python3.13/site-packages/sqlalchemy/orm/session.py", line 3680, in get
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     return self._get_impl(
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |            ~~~~~~~~~~~~~~^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |         entity,
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |         ^^^^^^^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     ...<7 lines>...
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |         bind_arguments=bind_arguments,
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     )
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     ^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |   File "/opt/airflow/.local/lib/python3.13/site-packages/sqlalchemy/orm/session.py", line 3859, in _get_impl
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     return db_load_fn(
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |         self,
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     ...<5 lines>...
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |         bind_arguments=bind_arguments,
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     )
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |   File "/opt/airflow/.local/lib/python3.13/site-packages/sqlalchemy/orm/loading.py", line 695, in load_on_pk_identity
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     session.execute(
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     ~~~~~~~~~~~~~~~^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |         q,
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |         ^^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     ...<2 lines>...
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |         bind_arguments=bind_arguments,
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     )
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     ^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |   File "/opt/airflow/.local/lib/python3.13/site-packages/sqlalchemy/orm/session.py", line 2351, in execute
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     return self._execute_internal(
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |            ~~~~~~~~~~~~~~~~~~~~~~^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |         statement,
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |         ^^^^^^^^^^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     ...<4 lines>...
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |         _add_event=_add_event,
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |         ^^^^^^^^^^^^^^^^^^^^^^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     )
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     ^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |   File "/opt/airflow/.local/lib/python3.13/site-packages/sqlalchemy/orm/session.py", line 2249, in _execute_internal
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     result: Result[Any] = compile_state_cls.orm_execute_statement(
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |                           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |         self,
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |         ^^^^^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     ...<4 lines>...
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |         conn,
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |         ^^^^^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     )
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     ^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |   File "/opt/airflow/.local/lib/python3.13/site-packages/sqlalchemy/orm/context.py", line 306, in orm_execute_statement
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     result = conn.execute(
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |         statement, params or {}, execution_options=execution_options
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     )
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |   File "/opt/airflow/.local/lib/python3.13/site-packages/sqlalchemy/engine/base.py", line 1419, in execute
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     return meth(
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |         self,
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |         distilled_parameters,
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |         execution_options or NO_OPTIONS,
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     )
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |   File "/opt/airflow/.local/lib/python3.13/site-packages/sqlalchemy/sql/elements.py", line 526, in _execute_on_connection
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     return connection._execute_clauseelement(
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |         self, distilled_params, execution_options
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     )
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     ^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |   File "/opt/airflow/.local/lib/python3.13/site-packages/sqlalchemy/engine/base.py", line 1641, in _execute_clauseelement
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     ret = self._execute_context(
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |         dialect,
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     ...<8 lines>...
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |         cache_hit=cache_hit,
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     )
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |   File "/opt/airflow/.local/lib/python3.13/site-packages/sqlalchemy/engine/base.py", line 1846, in _execute_context
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     return self._exec_single_context(
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |            ~~~~~~~~~~~~~~~~~~~~~~~~~^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |         dialect, context, statement, parameters
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     )
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     ^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |   File "/opt/airflow/.local/lib/python3.13/site-packages/sqlalchemy/engine/base.py", line 1986, in _exec_single_context
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     self._handle_dbapi_exception(
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |         e, str_statement, effective_parameters, cursor, context
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     )
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     ^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |   File "/opt/airflow/.local/lib/python3.13/site-packages/sqlalchemy/engine/base.py", line 2355, in _handle_dbapi_exception
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |   File "/opt/airflow/.local/lib/python3.13/site-packages/sqlalchemy/engine/base.py", line 1967, in _exec_single_context
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     self.dialect.do_execute(
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     ~~~~~~~~~~~~~~~~~~~~~~~^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |         cursor, str_statement, effective_parameters, context
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     )
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     ^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |   File "/opt/airflow/.local/lib/python3.13/site-packages/sqlalchemy/engine/default.py", line 951, in do_execute
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     cursor.execute(statement, parameters)
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |   File "/opt/airflow/.local/lib/python3.13/site-packages/MySQLdb/cursors.py", line 179, in execute
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     res = self._query(mogrified_query)
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |   File "/opt/airflow/.local/lib/python3.13/site-packages/MySQLdb/cursors.py", line 330, in _query
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     db.query(q)
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     ~~~~~~~~^^^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |   File "/opt/airflow/.local/lib/python3.13/site-packages/MySQLdb/connections.py", line 280, in query
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     _mysql.connection.query(self, query)
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     | sqlalchemy.exc.OperationalError: (MySQLdb.OperationalError) (4031, 'The client was disconnected by the server because of inactivity. See wait_timeout and interactive_timeout for configuring this behavior.')
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     | [SQL: SELECT ab_user.id AS ab_user_id, ab_user.first_name AS ab_user_first_name, ab_user.last_name AS ab_user_last_name, ab_user.username AS ab_user_username, ab_user.password AS ab_user_password, ab_user.active AS ab_user_active, ab_user.email AS ab_user_email, ab_user.last_login AS ab_user_last_login, ab_user.login_count AS ab_user_login_count, ab_user.fail_login_count AS ab_user_fail_login_count, ab_user.created_on AS ab_user_created_on, ab_user.changed_on AS ab_user_changed_on, ab_user.created_by_fk AS ab_user_created_by_fk, ab_user.changed_by_fk AS ab_user_changed_by_fk
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     | FROM ab_user
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     | WHERE ab_user.id = %s]
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     | [parameters: (1,)]
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     | (Background on this error at: https://sqlalche.me/e/20/e3q8)
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     | During handling of the above exception, another exception occurred:
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     | Traceback (most recent call last):
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |   File "/opt/airflow/.local/lib/python3.13/site-packages/starlette/middleware/wsgi.py", line 109, in __call__
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     await anyio.to_thread.run_sync(self.wsgi, environ, self.start_response)
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |   File "/opt/airflow/.local/lib/python3.13/site-packages/anyio/to_thread.py", line 56, in run_sync
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     return await get_async_backend().run_sync_in_worker_thread(
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |         func, args, abandon_on_cancel=abandon_on_cancel, limiter=limiter
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     )
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     ^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |   File "/opt/airflow/.local/lib/python3.13/site-packages/anyio/_backends/_asyncio.py", line 2485, in run_sync_in_worker_thread
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     return await future
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |            ^^^^^^^^^^^^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |   File "/opt/airflow/.local/lib/python3.13/site-packages/anyio/_backends/_asyncio.py", line 976, in run
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     result = context.run(func, *args)
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |   File "/opt/airflow/.local/lib/python3.13/site-packages/starlette/middleware/wsgi.py", line 147, in wsgi
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     for chunk in self.app(environ, start_response):
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |                  ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |   File "/opt/airflow/.local/lib/python3.13/site-packages/flask/app.py", line 2552, in __call__
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     return self.wsgi_app(environ, start_response)
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |            ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |   File "/opt/airflow/.local/lib/python3.13/site-packages/flask/app.py", line 2532, in wsgi_app
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     response = self.handle_exception(e)
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |   File "/opt/airflow/.local/lib/python3.13/site-packages/flask/app.py", line 1727, in handle_exception
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     server_error = self.ensure_sync(handler)(server_error)
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |   File "/opt/airflow/.local/lib/python3.13/site-packages/airflow/providers/fab/www/views.py", line 82, in show_traceback
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     return render_template("airflow/traceback.html"), 500
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |            ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |   File "/opt/airflow/.local/lib/python3.13/site-packages/flask/templating.py", line 147, in render_template
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     return _render(app, template, context)
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |   File "/opt/airflow/.local/lib/python3.13/site-packages/flask/templating.py", line 128, in _render
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     app.update_template_context(context)
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |   File "/opt/airflow/.local/lib/python3.13/site-packages/flask/app.py", line 997, in update_template_context
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     context.update(func())
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |                    ~~~~^^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |   File "/opt/airflow/.local/lib/python3.13/site-packages/flask_login/utils.py", line 405, in _user_context_processor
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     return dict(current_user=_get_user())
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |                              ~~~~~~~~~^^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |   File "/opt/airflow/.local/lib/python3.13/site-packages/flask_login/utils.py", line 370, in _get_user
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     current_app.login_manager._load_user()
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |   File "/opt/airflow/.local/lib/python3.13/site-packages/flask_login/login_manager.py", line 364, in _load_user
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     user = self._user_callback(user_id)
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |   File "/opt/airflow/.local/lib/python3.13/site-packages/airflow/providers/fab/auth_manager/security_manager/override.py", line 1384, in load_user
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     user = self.get_user_by_id(int(pk))
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |   File "/opt/airflow/.local/lib/python3.13/site-packages/airflow/providers/fab/auth_manager/security_manager/override.py", line 1390, in get_user_by_id
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     return self.session.get(self.user_model, pk)
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |            ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |   File "/opt/airflow/.local/lib/python3.13/site-packages/sqlalchemy/orm/session.py", line 3680, in get
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     return self._get_impl(
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |            ~~~~~~~~~~~~~~^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |         entity,
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |         ^^^^^^^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     ...<7 lines>...
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |         bind_arguments=bind_arguments,
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     )
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     ^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |   File "/opt/airflow/.local/lib/python3.13/site-packages/sqlalchemy/orm/session.py", line 3859, in _get_impl
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     return db_load_fn(
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |         self,
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     ...<5 lines>...
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |         bind_arguments=bind_arguments,
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     )
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |   File "/opt/airflow/.local/lib/python3.13/site-packages/sqlalchemy/orm/loading.py", line 695, in load_on_pk_identity
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     session.execute(
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     ~~~~~~~~~~~~~~~^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |         q,
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |         ^^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     ...<2 lines>...
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |         bind_arguments=bind_arguments,
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     )
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     ^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |   File "/opt/airflow/.local/lib/python3.13/site-packages/sqlalchemy/orm/session.py", line 2351, in execute
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     return self._execute_internal(
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |            ~~~~~~~~~~~~~~~~~~~~~~^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |         statement,
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |         ^^^^^^^^^^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     ...<4 lines>...
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |         _add_event=_add_event,
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |         ^^^^^^^^^^^^^^^^^^^^^^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     )
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     ^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |   File "/opt/airflow/.local/lib/python3.13/site-packages/sqlalchemy/orm/session.py", line 2249, in _execute_internal
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     result: Result[Any] = compile_state_cls.orm_execute_statement(
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |                           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |         self,
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |         ^^^^^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     ...<4 lines>...
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |         conn,
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |         ^^^^^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     )
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     ^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |   File "/opt/airflow/.local/lib/python3.13/site-packages/sqlalchemy/orm/context.py", line 306, in orm_execute_statement
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     result = conn.execute(
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |         statement, params or {}, execution_options=execution_options
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     )
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |   File "/opt/airflow/.local/lib/python3.13/site-packages/sqlalchemy/engine/base.py", line 1419, in execute
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     return meth(
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |         self,
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |         distilled_parameters,
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |         execution_options or NO_OPTIONS,
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     )
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |   File "/opt/airflow/.local/lib/python3.13/site-packages/sqlalchemy/sql/elements.py", line 526, in _execute_on_connection
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     return connection._execute_clauseelement(
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |         self, distilled_params, execution_options
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     )
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     ^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |   File "/opt/airflow/.local/lib/python3.13/site-packages/sqlalchemy/engine/base.py", line 1641, in _execute_clauseelement
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     ret = self._execute_context(
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |         dialect,
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     ...<8 lines>...
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |         cache_hit=cache_hit,
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     )
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |   File "/opt/airflow/.local/lib/python3.13/site-packages/sqlalchemy/engine/base.py", line 1813, in _execute_context
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     conn = self._revalidate_connection()
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |   File "/opt/airflow/.local/lib/python3.13/site-packages/sqlalchemy/engine/base.py", line 673, in _revalidate_connection
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     self._invalid_transaction()
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     ~~~~~~~~~~~~~~~~~~~~~~~~~^^
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |   File "/opt/airflow/.local/lib/python3.13/site-packages/sqlalchemy/engine/base.py", line 663, in _invalid_transaction
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     raise exc.PendingRollbackError(
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     ...<4 lines>...
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     |     )
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     | sqlalchemy.exc.PendingRollbackError: Can't reconnect until invalid transaction is rolled back.  Please rollback() fully before proceeding (Background on this error at: https://sqlalche.me/e/20/8s2b)
Oct 29 15:00:29 integration-airflow-new.mht python3.13[859125]:     +------------------------------------

    We got frustrated and blew away the database using the airflow db reset command - it deleted the db and kicked off the migrate.
    Every time the migration will get to this point and freeze: (we have mysql query logging enabled)

    2025-10-29T22:39:59.025609Z      3264 Query     SET time_zone = '+00:00'
2025-10-29T22:39:59.026833Z      3264 Query     DESCRIBE `airflow`.`alembic_version_fab`
2025-10-29T22:39:59.027421Z      3264 Query     DESCRIBE `airflow`.`alembic_version_fab`
2025-10-29T22:39:59.028196Z      3264 Query     CREATE TABLE alembic_version_fab (
        version_num VARCHAR(32) NOT NULL,
        CONSTRAINT alembic_version_fab_pkc PRIMARY KEY (version_num)
)

    When we eventually kill the migration, the slow query spits out an error indicating that query took "however long" to complete.

  7. dougforpres commented on Oct 29, 2025

    @dougforpres
    dougforpres
    on Oct 29, 2025

    PS: the missing time up until 15:00 today is due to Windows deciding to reboot my machine last night, so it was offline for that period

  8. dougforpres commented on Oct 30, 2025

    @dougforpres
    dougforpres
    on Oct 30, 2025

    We reinstalled everything using python 3.12 and at this point it seems to be behaving much better - we haven't retried a clean install on 3.13 yet.

  9. rebel-wansoo commented on Oct 30, 2025

    @rebel-wansoo
    rebel-wansoo
    on Oct 30, 2025
    Author

    It is obvious that if you restart the api-server the problem disappears “temporarily.” And after a few hours the problem happens again.

  10. ktaborski commented on Oct 31, 2025

    @ktaborski
    ktaborski
    on Oct 31, 2025

    same here:

    Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1910, in _execute_context
    self.dialect.do_execute(
  File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
    cursor.execute(statement, parameters)
psycopg2.OperationalError: SSL connection has been closed unexpectedly


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.12/site-packages/flask/app.py", line 2529, in wsgi_app
    response = self.full_dispatch_request()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/flask/app.py", line 1825, in full_dispatch_request
    rv = self.handle_user_exception(e)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/flask/app.py", line 1823, in full_dispatch_request
    rv = self.dispatch_request()
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/flask/app.py", line 1799, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/flask_appbuilder/security/views.py", line 713, in oauth_authorized
    user = self.appbuilder.sm.auth_user_oauth(userinfo)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/fab/auth_manager/security_manager/override.py", line 2031, in auth_user_oauth
    user = self.find_user(username=username)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/fab/auth_manager/security_manager/override.py", line 1426, in find_user
    return self.session.scalars(
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 1778, in scalars
    return self.execute(
           ^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 1717, in execute
    result = conn._execute_20(statement, params or {}, execution_options)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1710, in _execute_20
    return meth(self, args_10style, kwargs_10style, execution_options)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/sql/elements.py", line 334, in _execute_on_connection
    return connection._execute_clauseelement(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1577, in _execute_clauseelement
    ret = self._execute_context(
          ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1953, in _execute_context
    self._handle_dbapi_exception(
  File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 2134, in _handle_dbapi_exception
    util.raise_(
  File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/util/compat.py", line 211, in raise_
    raise exception
  File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1910, in _execute_context
    self.dialect.do_execute(
  File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) SSL connection has been closed unexpectedly

[SQL: SELECT ab_user.id, ab_user.first_name, ab_user.last_name, ab_user.username, ab_user.password, ab_user.active, ab_user.email, ab_user.last_login, ab_user.login_count, ab_user.fail_login_count, ab_user.created_on, ab_user.changed_on, ab_user.created_by_fk, ab_user.changed_by_fk 
FROM ab_user 
WHERE lower(ab_user.username) = lower(%(lower_1)s)]
[parameters: {'lower_1': 'ktaborski@xxx.com'}]
(Background on this error at: https://sqlalche.me/e/14/e3q8)
INFO:     172.16.109.9:0 - "GET /auth/oauth-authorized/okta?code=XXX&state=XXX HTTP/1.1" 500 Internal Server Error
Error authorizing OAuth access token: mismatching_state: CSRF Warning! State not equal in request and response.
2025-10-30T15:25:35.100979Z [error    ] Error authorizing OAuth access token: mismatching_state: CSRF Warning! State not equal in request and response. [flask_appbuilder.security.views] loc=views.py:684
INFO:     172.16.109.9:0 - "GET /auth/oauth-authorized/okta?code=XXX&state=XXX HTTP/1.1" 302 Found
INFO:     172.16.109.9:0 - "GET /auth/static/appbuilder/css/bootstrap.min.css HTTP/1.1" 304 Not Modified
INFO:     172.16.109.9:0 - "GET /auth/static/appbuilder/css/fontawesome/brands.min.css HTTP/1.1" 304 Not Modified
INFO:     172.16.109.9:0 - "GET /auth/static/appbuilder/css/fontawesome/solid.min.css HTTP/1.1" 304 Not Modified
INFO:     172.16.109.9:0 - "GET /auth/static/appbuilder/css/flags/flags16.css HTTP/1.1" 304 Not Modified
INFO:     172.16.109.9:0 - "GET /auth/static/appbuilder/css/fontawesome/v4-shims.min.css HTTP/1.1" 304 Not Modified
INFO:     172.16.109.9:0 - "GET /auth/static/dist/materialIcons.3e67dd6fbfcc4f3b5105.css HTTP/1.1" 304 Not Modified
INFO:     172.16.109.9:0 - "GET /auth/static/appbuilder/js/bootstrap.min.js HTTP/1.1" 304 Not Modified
INFO:     172.16.109.9:0 - "GET /auth/static/dist/flash.5583a9e0cf11f2be93da.css HTTP/1.1" 304 Not Modified
INFO:     172.16.109.9:0 - "GET /auth/static/appbuilder/js/ab_actions.js HTTP/1.1" 304 Not Modified
INFO:     172.16.109.9:0 - "GET /auth/static/dist/airflowDefaultTheme.ff5a35f322070b094aa2.css HTTP/1.1" 304 Not Modified
INFO:     172.16.109.9:0 - "GET /auth/static/dist/main.3cf3be1a0c5439bb640d.js HTTP/1.1" 304 Not Modified
INFO:     172.16.109.9:0 - "GET /auth/static/dist/main.3cf3be1a0c5439bb640d.css HTTP/1.1" 304 Not Modified
INFO:     172.16.109.9:0 - "GET /auth/static/dist/moment.9baee5ec3d7639a10897.js HTTP/1.1" 304 Not Modified
INFO:     172.16.109.9:0 - "GET /auth/static/appbuilder/css/webfonts/fa-solid-900.woff2 HTTP/1.1" 304 Not Modified
2025-10-30T15:25:38.884531Z [error    ] Exception on /oauth-authorized/okta [GET] [airflow.providers.fab.www.app] loc=app.py:1744
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.12/site-packages/flask/app.py", line 2529, in wsgi_app
    response = self.full_dispatch_request()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/flask/app.py", line 1825, in full_dispatch_request
    rv = self.handle_user_exception(e)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/flask/app.py", line 1823, in full_dispatch_request
    rv = self.dispatch_request()
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/flask/app.py", line 1799, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/flask_appbuilder/security/views.py", line 713, in oauth_authorized
    user = self.appbuilder.sm.auth_user_oauth(userinfo)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/fab/auth_manager/security_manager/override.py", line 2031, in auth_user_oauth
    user = self.find_user(username=username)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/fab/auth_manager/security_manager/override.py", line 1426, in find_user
    return self.session.scalars(
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 1778, in scalars
    return self.execute(
           ^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 1717, in execute
    result = conn._execute_20(statement, params or {}, execution_options)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1710, in _execute_20
    return meth(self, args_10style, kwargs_10style, execution_options)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/sql/elements.py", line 334, in _execute_on_connection
    return connection._execute_clauseelement(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1577, in _execute_clauseelement
    ret = self._execute_context(
          ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1808, in _execute_context
    conn = self._revalidate_connection()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 650, in _revalidate_connection
    self._invalid_transaction()
  File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 622, in _invalid_transaction
    raise exc.PendingRollbackError(
sqlalchemy.exc.PendingRollbackError: Can't reconnect until invalid transaction is rolled back. (Background on this error at: https://sqlalche.me/e/14/8s2b)

    The only solution for now is to restart api-server

  11. ShumzZzZz commented on Nov 2, 2025

    @ShumzZzZz
    ShumzZzZz
    on Nov 2, 2025

    same here, and restarting api-server pod (Im deploying using Helm) only works for a few hours, then same erro again, for me, when this happens, I cannot login, cannot access any pages under the Security section on the left hand side.

  12. 26 remaining items

  13. boudey commented on Jan 14, 2026

    @boudey
    boudey
    on Jan 14, 2026

    I am confirming the issue IS reproducible with apache-airflow-providers-fab==3.1.0

  14. Komalpawar22 commented on Jan 15, 2026

    @Komalpawar22
    Komalpawar22
    on Jan 15, 2026

    @here It worked for me with the fab provider version 3.0.2

  15. s1ny1998 commented on Jan 20, 2026

    @s1ny1998
    s1ny1998
    on Jan 20, 2026

    I am confirming the issue IS reproducible with apache-airflow-providers-fab==3.1.1 too

  16. s1ny1998 commented on Jan 20, 2026

    @s1ny1998
    s1ny1998
    on Jan 20, 2026

    @Komalpawar22 Did you downgrade your fab provider version?

  17. Komalpawar22 commented on Jan 20, 2026

    @Komalpawar22
    Komalpawar22
    on Jan 20, 2026

    @s1ny1998 Yes, using apache-airflow-providers-fab==3.0.2 now, which is working for me for now.

  18. vincbeck commented on Jan 20, 2026

    @vincbeck
    vincbeck
    on Jan 20, 2026
    Contributor

    It should be fixed with the latest version 3.1.2. Can someone test it?

  19. vincbeck commented on Jan 20, 2026

    @vincbeck
    vincbeck
    on Jan 20, 2026
    Contributor

    As per #59349, it is fixed

  20. Komalpawar22 commented on Jan 21, 2026

    @Komalpawar22
    Komalpawar22
    on Jan 21, 2026

    It should be fixed with the latest version 3.1.2. Can someone test it?

    I will test it.

  21. giannisyp commented on Jan 21, 2026

    @giannisyp
    giannisyp
    on Jan 21, 2026

    Is there a way to reproduce the error instead of waiting to reappear after few days ?

  22. anandsummer28 commented on Feb 16, 2026

    @anandsummer28
    anandsummer28
    on Feb 16, 2026

    Is there a way to reproduce the error instead of waiting to reappear after few days ?

    Posting this in case it helps others hit by the same error. We saw PendingRollbackError on login/UI refresh/logout with Airflow 3.1.1 (API server). Below: how we reproduced it and what fixed it.

    Environment: Airflow 3.1.1, API server on Kubernetes (same behavior should apply to non-K8s if you can set pool recycle and terminate DB connections).

    How to reproduce

    1. Shorten the DB pool recycle so connections age quickly (replace default with your namespace if needed):

      kubectl set env deployment/airflow-api-server \
  AIRFLOW__DATABASE__SQL_ALCHEMY_POOL_RECYCLE=30 \
  -n default

      Wait for the rollout to finish.

    2. Generate load and "idle in transaction" connections by calling the auth token API in a loop (replace YOUR_AIRFLOW_URL with your API base URL):

      AIRFLOW_URL="YOUR_AIRFLOW_URL"   # e.g. https://your-elb.amazonaws.com
for i in $(seq 1 80); do
  CODE=$(curl -s -o /dev/null -w "%{http_code}" -X POST "$AIRFLOW_URL/auth/token" \
    -H "Content-Type: application/json" \
    -d '{"username": "admin", "password": "YOURPASSWORD"}')
  echo "$i $CODE"
  sleep 0.5
done

      You’ll typically see 201 for most requests.

    3. Connect to the Airflow metadata database (Postgres) and list connections:

      SELECT pid, usename, application_name, state, state_change
FROM pg_stat_activity
WHERE datname = 'airflow';

      You should see some idle and/or idle in transaction rows from the app.

    4. Terminate both idle and idle-in-transaction backends (this leaves your current psql session alone):

      SELECT pg_terminate_backend(pid)
FROM pg_stat_activity
WHERE datname = 'airflow'
  AND state IN ('idle', 'idle in transaction')
  AND pid <> pg_backend_pid();

    5. Trigger the error:

      • Run the token loop again and/or
      • In the Airflow UI (while already logged in), refresh the page or click Logout.

      You should see 500 Internal Server Error and in the API server logs: PendingRollbackError and stack traces in airflow.providers.fab.www.app / security_manager / get_user_by_id. The UI can stay broken until the pod is restarted.

    Note: Terminating only idle in transaction was not enough for us; terminating both idle and idle in transaction made the failure reproducible.

    Fix that worked: FAB 3.0.2 + pool settings

    1. Pin the FAB provider to apache-airflow-providers-fab==3.0.2 and ensure it is used at API server (and webserver, if used) startup.
    2. Set DB pool options:
      AIRFLOW__DATABASE__SQL_ALCHEMY_POOL_PRE_PING=True and
      AIRFLOW__DATABASE__SQL_ALCHEMY_POOL_RECYCLE=1800

    After applying this, the same reproduction steps either no longer hit the error or, when a 500 occurred, the UI recovered on the next request instead of staying stuck.

    Example for the official Airflow Helm chart (override apiServer.args / apiServer.env and same for webserver if you use it):

    airflow:
  config:
    fab:
      enable_proxy_fix: "True"
      session_lifetime_minutes: "1440"

  apiServer:
    args:
      - bash
      - "-c"
      - |
        pip install --no-cache-dir "apache-airflow-providers-fab==3.0.2" \
        && exec airflow api-server
    env:
      - name: AIRFLOW__DATABASE__SQL_ALCHEMY_POOL_PRE_PING
        value: "True"
      - name: AIRFLOW__DATABASE__SQL_ALCHEMY_POOL_RECYCLE
        value: "1800"

  webserver:   # if you use the webserver
    args:
      - bash
      - "-c"
      - |
        pip install --no-cache-dir "apache-airflow-providers-fab==3.0.2" \
        && exec airflow webserver
    env:
      - name: AIRFLOW__DATABASE__SQL_ALCHEMY_POOL_PRE_PING
        value: "True"
      - name: AIRFLOW__DATABASE__SQL_ALCHEMY_POOL_RECYCLE
        value: "1800"

    The session-cleanup fixes for this issue are in apache-airflow-providers-fab 3.1.2+. If problems persist on 3.0.2, try >=3.1.2 (e.g. 3.1.2 or 3.2.0) with the same pool settings.

    Summary

    Step Action
    Reproduce Set POOL_RECYCLE=30, hammer /auth/token, then in Postgres terminate idle and idle in transaction backends, then refresh UI or logout.
    Fix Use apache-airflow-providers-fab==3.0.2 (or >=3.1.2) at startup + PRE_PING=True and POOL_RECYCLE=1800.

    Hope this helps anyone else hitting the same error.

  23. mwullink commented on Mar 5, 2026

    @mwullink
    mwullink
    on Mar 5, 2026

    still getting this error with airflow 3.1.7 and apache-airflow-providers-fab>=3.4.0
    setting the AIRFLOW__DATABASE__SQL_ALCHEMY_POOL_PRE_PING and AIRFLOW__DATABASE__SQL_ALCHEMY_POOL_RECYCLE does not fix it for me.

  24. sachin-dixit-wynk commented on Mar 6, 2026

    @sachin-dixit-wynk
    sachin-dixit-wynk
    on Mar 6, 2026

    still getting this with 3.1.7.

    =================================================

    INFO: 127.0.0.6:52475 - "GET / HTTP/1.1" 200 OK
    INFO: 127.0.0.6:52335 - "GET / HTTP/1.1" 200 OK
    INFO: 127.0.0.6:48085 - "GET /dags/dp-dev-daily-end-workflow/runs/scheduled__2026-03-05T02%3A30%3A00%2B00%3A00/tasks/dp-dev-wait-timer-end-daily-workflow HTTP/1.1" 500 Internal Server Error
    ERROR: Exception in ASGI application
    Traceback (most recent call last):
    File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1967, in _exec_single_context
    self.dialect.do_execute(
    File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/default.py", line 952, in do_execute
    cursor.execute(statement, parameters)
    psycopg2.OperationalError: SSL SYSCALL error: EOF detected

    The above exception was the direct cause of the following exception:

    Traceback (most recent call last):
    File "/home/airflow/.local/lib/python3.12/site-packages/uvicorn/protocols/http/httptools_impl.py", line 416, in run_asgi
    result = await app( # type: ignore[func-returns-value]
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/home/airflow/.local/lib/python3.12/site-packages/fastapi/applications.py", line 1082, in call
    await super().call(scope, receive, send)
    File "/home/airflow/.local/lib/python3.12/site-packages/starlette/applications.py", line 113, in call
    await self.middleware_stack(scope, receive, send)
    File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/errors.py", line 186, in call
    raise exc
    File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/errors.py", line 164, in call
    await self.app(scope, receive, _send)
    File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/gzip.py", line 29, in call
    await responder(scope, receive, send)
    File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/gzip.py", line 130, in call
    await super().call(scope, receive, send)
    File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/gzip.py", line 46, in call
    await self.app(scope, receive, self.send_with_compression)
    File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/cors.py", line 85, in call
    await self.app(scope, receive, send)
    File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/base.py", line 182, in call
    with recv_stream, send_stream, collapse_excgroups():
    ^^^^^^^^^^^^^^^^^^^^
    File "/usr/python/lib/python3.12/contextlib.py", line 158, in exit
    self.gen.throw(value)
    File "/home/airflow/.local/lib/python3.12/site-packages/starlette/_utils.py", line 85, in collapse_excgroups
    raise exc
    File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/base.py", line 184, in call
    response = await self.dispatch_func(request, call_next)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/home/airflow/.local/lib/python3.12/site-packages/airflow/api_fastapi/auth/middlewares/refresh_token.py", line 49, in dispatch
    new_user, current_user = await self._refresh_user(current_token)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/home/airflow/.local/lib/python3.12/site-packages/airflow/api_fastapi/auth/middlewares/refresh_token.py", line 80, in _refresh_user
    user = await resolve_user_from_token(current_token)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/home/airflow/.local/lib/python3.12/site-packages/airflow/api_fastapi/core_api/security.py", line 100, in resolve_user_from_token
    return await get_auth_manager().get_user_from_token(token_str)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/home/airflow/.local/lib/python3.12/site-packages/airflow/api_fastapi/auth/managers/base_auth_manager.py", line 111, in get_user_from_token
    return self.deserialize_user(payload)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/home/airflow/.local/lib/python3.12/site-packages/cachetools/_cachedmethod.py", line 375, in call
    return wrapper(self._obj, *args, **kwargs)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/home/airflow/.local/lib/python3.12/site-packages/cachetools/_cachedmethod.py", line 358, in wrapper
    v = method(self, *args, **kwargs)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/fab/auth_manager/fab_auth_manager.py", line 267, in deserialize_user
    return self.session.scalars(select(User).where(User.id == int(token["sub"]))).one()
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/scoping.py", line 1891, in scalars
    return self._proxied.scalars(
    ^^^^^^^^^^^^^^^^^^^^^^
    File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2459, in scalars
    return self._execute_internal(
    ^^^^^^^^^^^^^^^^^^^^^^^
    File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2249, in _execute_internal
    result: Result[Any] = compile_state_cls.orm_execute_statement(
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/context.py", line 306, in orm_execute_statement
    result = conn.execute(
    ^^^^^^^^^^^^^
    File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1419, in execute
    return meth(
    ^^^^^
    File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/sql/elements.py", line 527, in _execute_on_connection
    return connection._execute_clauseelement(
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1641, in _execute_clauseelement
    ret = self._execute_context(
    ^^^^^^^^^^^^^^^^^^^^^^
    File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1846, in _execute_context
    return self._exec_single_context(
    ^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1986, in _exec_single_context
    self._handle_dbapi_exception(
    File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 2363, in _handle_dbapi_exception
    raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
    File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1967, in _exec_single_context
    self.dialect.do_execute(
    File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/default.py", line 952, in do_execute
    cursor.execute(statement, parameters)
    sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) SSL SYSCALL error: EOF detected

    [SQL: SELECT ab_user.id, ab_user.first_name, ab_user.last_name, ab_user.username, ab_user.password, ab_user.active, ab_user.email, ab_user.last_login, ab_user.login_count, ab_user.fail_login_count, ab_user.created_on, ab_user.changed_on, ab_user.created_by_fk, ab_user.changed_by_fk
    FROM ab_user
    WHERE ab_user.id = %(id_1)s]
    [parameters: {'id_1': 1}]
    (Background on this error at: https://sqlalche.me/e/20/e3q8)
    INFO: 127.0.0.6:38111 - "GET /api/v2/dags/dp-dev-daily-end-workflow/dagRuns/scheduled__2026-03-05T02%3A30%3A00%2B00%3A00/taskInstances/dp-dev-wait-timer-end-daily-workflow/links?map_index=-1 HTTP/1.1" 500 Internal Server Error
    ERROR: Exception in ASGI application

    • Exception Group Traceback (most recent call last):
      | File "/home/airflow/.local/lib/python3.12/site-packages/starlette/_utils.py", line 79, in collapse_excgroups
      | yield
      | File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/base.py", line 183, in call
      | async with anyio.create_task_group() as task_group:
      | ^^^^^^^^^^^^^^^^^^^^^^^^^
      | File "/home/airflow/.local/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 783, in aexit
      | raise BaseExceptionGroup(
      | ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
      +-+---------------- 1 ----------------
      | Traceback (most recent call last):
      | File "/home/airflow/.local/lib/python3.12/site-packages/uvicorn/protocols/http/httptools_impl.py", line 416, in run_asgi
      | result = await app( # type: ignore[func-returns-value]
      | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      | File "/home/airflow/.local/lib/python3.12/site-packages/fastapi/applications.py", line 1082, in call
      | await super().call(scope, receive, send)
      | File "/home/airflow/.local/lib/python3.12/site-packages/starlette/applications.py", line 113, in call
      | await self.middleware_stack(scope, receive, send)
      | File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/errors.py", line 186, in call
      | raise exc
      | File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/errors.py", line 164, in call
      | await self.app(scope, receive, _send)
      | File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/gzip.py", line 29, in call
      | await responder(scope, receive, send)
      | File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/gzip.py", line 130, in call
      | await super().call(scope, receive, send)
      | File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/gzip.py", line 46, in call
      | await self.app(scope, receive, self.send_with_compression)
      | File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/cors.py", line 85, in call
      | await self.app(scope, receive, send)
      | File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/base.py", line 182, in call
      | with recv_stream, send_stream, collapse_excgroups():
      | ^^^^^^^^^^^^^^^^^^^^
      | File "/usr/python/lib/python3.12/contextlib.py", line 158, in exit
      | self.gen.throw(value)
      | File "/home/airflow/.local/lib/python3.12/site-packages/starlette/_utils.py", line 85, in collapse_excgroups
      | raise exc
      | File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/base.py", line 184, in call
      | response = await self.dispatch_func(request, call_next)
      | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      | File "/home/airflow/.local/lib/python3.12/site-packages/airflow/api_fastapi/auth/middlewares/refresh_token.py", line 49, in dispatch
      | new_user, current_user = await self._refresh_user(current_token)
      | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      | File "/home/airflow/.local/lib/python3.12/site-packages/airflow/api_fastapi/auth/middlewares/refresh_token.py", line 80, in _refresh_user
      | user = await resolve_user_from_token(current_token)
      | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      | File "/home/airflow/.local/lib/python3.12/site-packages/airflow/api_fastapi/core_api/security.py", line 100, in resolve_user_from_token
      | return await get_auth_manager().get_user_from_token(token_str)
      | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      | File "/home/airflow/.local/lib/python3.12/site-packages/airflow/api_fastapi/auth/managers/base_auth_manager.py", line 111, in get_user_from_token
      | return self.deserialize_user(payload)
      | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      | File "/home/airflow/.local/lib/python3.12/site-packages/cachetools/_cachedmethod.py", line 375, in call
      | return wrapper(self._obj, *args, **kwargs)
      | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      | File "/home/airflow/.local/lib/python3.12/site-packages/cachetools/_cachedmethod.py", line 358, in wrapper
      | v = method(self, *args, **kwargs)
      | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      | File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/fab/auth_manager/fab_auth_manager.py", line 267, in deserialize_user
      | return self.session.scalars(select(User).where(User.id == int(token["sub"]))).one()
      | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      | File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/scoping.py", line 1891, in scalars
      | return self._proxied.scalars(
      | ^^^^^^^^^^^^^^^^^^^^^^
      | File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2459, in scalars
      | return self._execute_internal(
      | ^^^^^^^^^^^^^^^^^^^^^^^
      | File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2249, in _execute_internal
      | result: Result[Any] = compile_state_cls.orm_execute_statement(
      | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      | File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/context.py", line 306, in orm_execute_statement
      | result = conn.execute(
      | ^^^^^^^^^^^^^
      | File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1419, in execute
      | return meth(
      | ^^^^^
      | File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/sql/elements.py", line 527, in _execute_on_connection
      | return connection._execute_clauseelement(
      | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      | File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1641, in _execute_clauseelement
      | ret = self._execute_context(
      | ^^^^^^^^^^^^^^^^^^^^^^
      | File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1813, in _execute_context
      | conn = self._revalidate_connection()
      | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      | File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 673, in _revalidate_connection
      | self._invalid_transaction()
      | File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 663, in _invalid_transaction
      | raise exc.PendingRollbackError(
      | sqlalchemy.exc.PendingRollbackError: Can't reconnect until invalid transaction is rolled back. Please rollback() fully before proceeding (Background on this error at: https://sqlalche.me/e/20/8s2b)
      +------------------------------------

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
    File "/home/airflow/.local/lib/python3.12/site-packages/uvicorn/protocols/http/httptools_impl.py", line 416, in run_asgi
    result = await app( # type: ignore[func-returns-value]
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/home/airflow/.local/lib/python3.12/site-packages/fastapi/applications.py", line 1082, in call
    await super().call(scope, receive, send)
    File "/home/airflow/.local/lib/python3.12/site-packages/starlette/applications.py", line 113, in call
    await self.middleware_stack(scope, receive, send)
    File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/errors.py", line 186, in call
    raise exc
    File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/errors.py", line 164, in call
    await self.app(scope, receive, _send)
    File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/gzip.py", line 29, in call
    await responder(scope, receive, send)
    File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/gzip.py", line 130, in call
    await super().call(scope, receive, send)
    File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/gzip.py", line 46, in call
    await self.app(scope, receive, self.send_with_compression)
    File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/cors.py", line 85, in call
    await self.app(scope, receive, send)
    File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/base.py", line 182, in call
    with recv_stream, send_stream, collapse_excgroups():
    ^^^^^^^^^^^^^^^^^^^^
    File "/usr/python/lib/python3.12/contextlib.py", line 158, in exit
    self.gen.throw(value)
    File "/home/airflow/.local/lib/python3.12/site-packages/starlette/_utils.py", line 85, in collapse_excgroups
    raise exc
    File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/base.py", line 184, in call
    response = await self.dispatch_func(request, call_next)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/home/airflow/.local/lib/python3.12/site-packages/airflow/api_fastapi/auth/middlewares/refresh_token.py", line 49, in dispatch
    new_user, current_user = await self._refresh_user(current_token)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/home/airflow/.local/lib/python3.12/site-packages/airflow/api_fastapi/auth/middlewares/refresh_token.py", line 80, in _refresh_user
    user = await resolve_user_from_token(current_token)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/home/airflow/.local/lib/python3.12/site-packages/airflow/api_fastapi/core_api/security.py", line 100, in resolve_user_from_token
    return await get_auth_manager().get_user_from_token(token_str)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/home/airflow/.local/lib/python3.12/site-packages/airflow/api_fastapi/auth/managers/base_auth_manager.py", line 111, in get_user_from_token
    return self.deserialize_user(payload)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/home/airflow/.local/lib/python3.12/site-packages/cachetools/_cachedmethod.py", line 375, in call
    return wrapper(self._obj, *args, **kwargs)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/home/airflow/.local/lib/python3.12/site-packages/cachetools/_cachedmethod.py", line 358, in wrapper
    v = method(self, *args, **kwargs)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/fab/auth_manager/fab_auth_manager.py", line 267, in deserialize_user
    return self.session.scalars(select(User).where(User.id == int(token["sub"]))).one()
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/scoping.py", line 1891, in scalars
    return self._proxied.scalars(
    ^^^^^^^^^^^^^^^^^^^^^^
    File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2459, in scalars
    return self._execute_internal(
    ^^^^^^^^^^^^^^^^^^^^^^^
    File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2249, in _execute_internal
    result: Result[Any] = compile_state_cls.orm_execute_statement(
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/context.py", line 306, in orm_execute_statement
    result = conn.execute(
    ^^^^^^^^^^^^^
    File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1419, in execute
    return meth(
    ^^^^^
    File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/sql/elements.py", line 527, in _execute_on_connection
    return connection._execute_clauseelement(
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1641, in _execute_clauseelement
    ret = self._execute_context(
    ^^^^^^^^^^^^^^^^^^^^^^
    File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1813, in _execute_context
    conn = self._revalidate_connection()
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 673, in _revalidate_connection
    self._invalid_transaction()
    File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 663, in _invalid_transaction
    raise exc.PendingRollbackError(
    sqlalchemy.exc.PendingRollbackError: Can't reconnect until invalid transaction is rolled back. Please rollback() fully before proceeding (Background on this error at: https://sqlalche.me/e/20/8s2b)
    INFO: 127.0.0.6:48657 - "GET /api/v2/dags/dp-dev-daily-end-workflow/dagRuns/scheduled__2026-03-05T02%3A30%3A00%2B00%3A00/taskInstances/dp-dev-wait-timer-end-daily-workflow/logs/1?map_index=-1 HTTP/1.1" 500 Internal Server Error
    ERROR: Exception in ASGI application

    • Exception Group Traceback (most recent call last):
      | File "/home/airflow/.local/lib/python3.12/site-packages/starlette/_utils.py", line 79, in collapse_excgroups
      | yield
      | File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/base.py", line 183, in call
      | async with anyio.create_task_group() as task_group:
      | ^^^^^^^^^^^^^^^^^^^^^^^^^
      | File "/home/airflow/.local/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 783, in aexit
      | raise BaseExceptionGroup(
      | ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
      +-+---------------- 1 ----------------
      | Traceback (most recent call last):
      | File "/home/airflow/.local/lib/python3.12/site-packages/uvicorn/protocols/http/httptools_impl.py", line 416, in run_asgi
      | result = await app( # type: ignore[func-returns-value]
      | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      | File "/home/airflow/.local/lib/python3.12/site-packages/fastapi/applications.py", line 1082, in call
      | await super().call(scope, receive, send)
      | File "/home/airflow/.local/lib/python3.12/site-packages/starlette/applications.py", line 113, in call
      | await self.middleware_stack(scope, receive, send)
      | File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/errors.py", line 186, in call
      | raise exc
      | File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/errors.py", line 164, in call
      | await self.app(scope, receive, _send)
      | File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/gzip.py", line 29, in call
      | await responder(scope, receive, send)
      | File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/gzip.py", line 130, in call
      | await super().call(scope, receive, send)
      | File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/gzip.py", line 46, in call
      | await self.app(scope, receive, self.send_with_compression)
      | File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/cors.py", line 85, in call
      | await self.app(scope, receive, send)
      | File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/base.py", line 182, in call
      | with recv_stream, send_stream, collapse_excgroups():
      | ^^^^^^^^^^^^^^^^^^^^
      | File "/usr/python/lib/python3.12/contextlib.py", line 158, in exit
      | self.gen.throw(value)
      | File "/home/airflow/.local/lib/python3.12/site-packages/starlette/_utils.py", line 85, in collapse_excgroups
      | raise exc
      | File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/base.py", line 184, in call
      | response = await self.dispatch_func(request, call_next)
      | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      | File "/home/airflow/.local/lib/python3.12/site-packages/airflow/api_fastapi/auth/middlewares/refresh_token.py", line 49, in dispatch
      | new_user, current_user = await self._refresh_user(current_token)
      | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      | File "/home/airflow/.local/lib/python3.12/site-packages/airflow/api_fastapi/auth/middlewares/refresh_token.py", line 80, in _refresh_user
      | user = await resolve_user_from_token(current_token)
      | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      | File "/home/airflow/.local/lib/python3.12/site-packages/airflow/api_fastapi/core_api/security.py", line 100, in resolve_user_from_token
      | return await get_auth_manager().get_user_from_token(token_str)
      | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      | File "/home/airflow/.local/lib/python3.12/site-packages/airflow/api_fastapi/auth/managers/base_auth_manager.py", line 111, in get_user_from_token
      | return self.deserialize_user(payload)
      | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      | File "/home/airflow/.local/lib/python3.12/site-packages/cachetools/_cachedmethod.py", line 375, in call
      | return wrapper(self._obj, *args, **kwargs)
      | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      | File "/home/airflow/.local/lib/python3.12/site-packages/cachetools/_cachedmethod.py", line 358, in wrapper
      | v = method(self, *args, **kwargs)
      | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      | File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/fab/auth_manager/fab_auth_manager.py", line 267, in deserialize_user
      | return self.session.scalars(select(User).where(User.id == int(token["sub"]))).one()
      | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      | File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/scoping.py", line 1891, in scalars
      | return self._proxied.scalars(
      | ^^^^^^^^^^^^^^^^^^^^^^
      | File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2459, in scalars
      | return self._execute_internal(
      | ^^^^^^^^^^^^^^^^^^^^^^^
      | File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2249, in _execute_internal
      | result: Result[Any] = compile_state_cls.orm_execute_statement(
      | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      | File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/context.py", line 306, in orm_execute_statement
      | result = conn.execute(
      | ^^^^^^^^^^^^^
      | File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1419, in execute
      | return meth(
      | ^^^^^
      | File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/sql/elements.py", line 527, in _execute_on_connection
      | return connection._execute_clauseelement(
      | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      | File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1641, in _execute_clauseelement
      | ret = self._execute_context(
      | ^^^^^^^^^^^^^^^^^^^^^^
      | File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1813, in _execute_context
      | conn = self._revalidate_connection()
      | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      | File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 673, in _revalidate_connection
      | self._invalid_transaction()
      | File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 663, in _invalid_transaction
      | raise exc.PendingRollbackError(
      | sqlalchemy.exc.PendingRollbackError: Can't reconnect until invalid transaction is rolled back. Please rollback() fully before proceeding (Background on this error at: https://sqlalche.me/e/20/8s2b)
      +------------------------------------

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
    File "/home/airflow/.local/lib/python3.12/site-packages/uvicorn/protocols/http/httptools_impl.py", line 416, in run_asgi
    result = await app( # type: ignore[func-returns-value]
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/home/airflow/.local/lib/python3.12/site-packages/fastapi/applications.py", line 1082, in call
    await super().call(scope, receive, send)
    File "/home/airflow/.local/lib/python3.12/site-packages/starlette/applications.py", line 113, in call
    await self.middleware_stack(scope, receive, send)
    File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/errors.py", line 186, in call
    raise exc
    File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/errors.py", line 164, in call
    await self.app(scope, receive, _send)
    File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/gzip.py", line 29, in call
    await responder(scope, receive, send)
    File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/gzip.py", line 130, in call
    await super().call(scope, receive, send)
    File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/gzip.py", line 46, in call
    await self.app(scope, receive, self.send_with_compression)
    File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/cors.py", line 85, in call
    await self.app(scope, receive, send)

  25. robertlindner commented on Apr 9, 2026

    @robertlindner
    robertlindner
    on Apr 9, 2026

    still getting this error with airflow 3.1.7 and apache-airflow-providers-fab>=3.4.0 setting the AIRFLOW__DATABASE__SQL_ALCHEMY_POOL_PRE_PING and AIRFLOW__DATABASE__SQL_ALCHEMY_POOL_RECYCLE does not fix it for me.

    We are also facing this issue in airflow 3.1.7 with apache-airflow-providers-fab 3.2.0

    I just checked the documentation, and apparently the proposed values are the defaults since quite some time:
    https://airflow.apache.org/docs/apache-airflow/2.3.4/configurations-ref.html#sql-alchemy-pool-pre-ping
    https://airflow.apache.org/docs/apache-airflow/2.3.4/configurations-ref.html#sql-alchemy-pool-recycle

  26. stefankeidel commented on Apr 28, 2026

    @stefankeidel
    stefankeidel
    on Apr 28, 2026
    Contributor

    I have to pile on here, 3.1.7 and FAB provider 3.2.0 and we just saw this this morning

  27. sklempna commented on May 28, 2026

    @sklempna
    sklempna
    on May 28, 2026

    I am also still experiencing this issue. Airflow is 3.1.7, apache-airflow-providers-fab is 3.2.0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    affected_version:3.1Issues Reported for 3.1area:corekind:bugThis is a clearly a bugneeds-triagelabel for new issues that we didn't triage yetpriority:upgrade_to_airflow3

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions