Skip to content

Recover the revoked-token check from a stale DB connection (fixes idle-timeout 500) - #71444

Open
uplsh580 wants to merge 1 commit into
apache:mainfrom
uplsh580:fix/revoked-token-stale-session-retry-71395
Open

Recover the revoked-token check from a stale DB connection (fixes idle-timeout 500)#71444
uplsh580 wants to merge 1 commit into
apache:mainfrom
uplsh580:fix/revoked-token-stale-session-retry-71395

Conversation

@uplsh580

@uplsh580 uplsh580 commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

Recover the JWT revoked-token check from a stale pooled DB connection so the first authenticated request after an idle period no longer returns HTTP 500.

Issue: #71395

Problem

With MySQL as the metadata DB and the FAB auth manager, the first authenticated request after the api-server sits idle longer than MySQL's wait_timeout fails with HTTP 500. The exception is an OperationalError (MySQL error 4031, "The client was disconnected by the server because of inactivity") raised from RevokedToken.is_revoked inside BaseAuthManager.get_user_from_token. The immediately following request succeeds, because the failed query resets the pooled connection — so in production it looks like intermittent 500s after overnight idle periods.

Root cause

  1. The FAB auth manager shares core's scoped settings.Session. A prior request (e.g. FabAuthManager.deserialize_user, or FAB's Flask views) can leave that scoped session bound to a connection the database later drops on idle timeout. Because the connection is never returned to the pool, there is no checkout eventpool_pre_ping / pool_recycle cannot help.
  2. This exact failure mode was reported in apache-airflow-providers-fab 3.4.0: Intermittent OperationalError (4031) when FAB session reuses a MySQL connection dropped due to idle timeout #62903 and fixed by fix(fab): recover from first idle MySQL disconnect in token auth #62919, which added a discard-and-retry recovery to FabAuthManager.deserialize_user.
  3. But 3.2.0 introduced the JWT revocation check (AIP-84 : Add JWT token revokation for logout invalidation (#47952) #61339 / AIP-84): RevokedToken.is_revoked in get_user_from_token now runs before deserialize_user, so it is the first thing to touch the poisoned scoped session — and it had no recovery logic. The 500 that fix(fab): recover from first idle MySQL disconnect in token auth #62919 fixed is back, just raised one step earlier in the auth path.

Fix

Give the revoked-token check the same recovery deserialize_user already has: on SQLAlchemyError, discard the scoped session (settings.Session.remove()) and retry the check once on a fresh connection. The recovery is factored into a small BaseAuthManager._is_token_revoked helper so get_user_from_token stays readable.

This is the per-call-site variant proposed in the issue (the one that has been running in production for several weeks and eliminated these 500s). It is minimal and mirrors the established #62919 pattern rather than introducing a new session-management abstraction. A persistent DB error still surfaces — only a single transient stale-connection error is absorbed.

Tests

Added two regression tests in test_base_auth_manager.py. Both raise a 4031-style OperationalError on the first is_revoked call and assert the scoped session is discarded (settings.Session.remove()) and the check is retried exactly once:

  • retry returns not revoked → the request is served transparently;
  • retry returns revokedInvalidTokenError("Token has been revoked") is still raised.

related: #71395


Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

Generated-by: Claude Code following the guidelines

@boring-cyborg boring-cyborg Bot added the area:API Airflow's REST/HTTP API label Aug 11, 2026
uplsh580 added a commit to uplsh580/airflow that referenced this pull request Aug 11, 2026
uplsh580 added a commit to uplsh580/airflow that referenced this pull request Aug 11, 2026
@uplsh580
uplsh580 force-pushed the fix/revoked-token-stale-session-retry-71395 branch from 9107fe5 to 6850a84 Compare August 11, 2026 16:06
@uplsh580
uplsh580 marked this pull request as ready for review August 11, 2026 16:06
@uplsh580
uplsh580 requested a review from vincbeck as a code owner August 11, 2026 16:06
Comment thread airflow-core/newsfragments/71444.bugfix.rst Outdated
@uplsh580
uplsh580 force-pushed the fix/revoked-token-stale-session-retry-71395 branch 2 times, most recently from 7781cc1 to 291c92a Compare August 11, 2026 23:28
Comment thread airflow-core/src/airflow/api_fastapi/auth/managers/base_auth_manager.py Outdated
@uplsh580
uplsh580 force-pushed the fix/revoked-token-stale-session-retry-71395 branch from 603906a to c486ed8 Compare August 12, 2026 13:23
The JWT revocation check added in 3.2.0 (RevokedToken.is_revoked in
BaseAuthManager.get_user_from_token) is the first DB access in the auth path
and runs on the shared scoped session. A prior request (e.g. FAB's
deserialize_user) can leave that session bound to a connection the database
later drops on idle timeout (MySQL error 4031, "disconnected ... because of
inactivity"). The connection is never re-checked-out, so pool_pre_ping /
pool_recycle cannot detect it, and the first authenticated request after an
idle period fails with HTTP 500.

deserialize_user already recovers from this class of failure (apache#62919), but the
revocation check runs earlier in get_user_from_token and had no recovery, so
the 500 simply returned one step sooner. Apply the same discard-and-retry: on
SQLAlchemyError, remove the poisoned scoped session and retry the check once on
a fresh connection.

Related to apache#71395
@uplsh580
uplsh580 force-pushed the fix/revoked-token-stale-session-retry-71395 branch from c486ed8 to f54e0ea Compare August 12, 2026 13:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:API Airflow's REST/HTTP API

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants