Repository navigation
ref(auth): Stop falling back to scopes bitfield in get_scopes - #127282
Conversation
Co-Authored-By: Shaun Kaasten <shaun.kaasten@sentry.io>
| if self.scope_list: | ||
| return sorted(self.scope_list) | ||
| return sorted([k for k, v in self.scopes.items() if v]) | ||
| return sorted(self.scope_list) |
There was a problem hiding this comment.
Is there a reason this model returns them sorted, but the other doesn't?
There was a problem hiding this comment.
No functional reason. The difference was already there before this PR. HasApiScopes.get_scopes() has always sorted, and its docstring promises alphabetical order. ApiGrant doesn't use the mixin and has its own get_scopes() that never sorted. I left both behaviors as they were so this PR only removes the fallback.
In practice the grant's order doesn't leak out. When a grant is exchanged, the resulting ApiToken goes through the pre_save hook, and add_scope_hierarchy sorts its scope_list. The seer agent-token callers turn the grant's scopes into sets. If we want the two models to match, sorting in ApiGrant.get_scopes() is a one-line change and would be easy to do in a follow-up.
There was a problem hiding this comment.
Gonna keep as is to avoid any unknown dependencies on that order.
Now that get_scopes() reads only scope_list (#127282), drop the legacy scopes BitField from HasApiScopes and ApiGrant. The columns are made nullable and moved to pending deletion with SafeRemoveField; a follow-up migration will drop them once this has deployed. Refs #50475 Refs ENG-3384 Co-Authored-By: Shaun Kaasten <shaun.kaasten@sentry.io> Co-Authored-By: shaun.kaasten <shaun.kaasten@sentry.io>
…127386) Phase 1 of 2 for dropping the legacy `scopes` BitField columns. Follow-up to #127282, which made `get_scopes()` read only `scope_list`. Refs #50475, ENG-3384 **What changed** - `src/sentry/models/apiscopes.py`: removed the `scopes` BitField and `ScopesDict` from `HasApiScopes`, including the `assert` that kept the bitfield keys in sync with `ApiScopes`. New scopes no longer need an `AlterField` migration. - `src/sentry/models/apigrant.py`: removed `ApiGrant`'s own `scopes` BitField. - `src/sentry/migrations/1215_remove_scopes_bitfield_pending.py`: makes `scopes` nullable, then `SafeRemoveField(..., MOVE_TO_PENDING)` on `apiauthorization`, `apigrant`, `apikey`, `apitoken`, `sentryapp`. - `src/sentry/hybridcloud/migrations/0044_remove_scopes_bitfield_pending.py`: same for `apikeyreplica` and `apitokenreplica`. - Tests: removed direct `scopes=` bitfield use from `test_apitoken.py`, `test_apiapplication.py` (deletions), and `test_sentry_app_updater.py`. **Deploy safety** - The only SQL is `ALTER COLUMN "scopes" DROP NOT NULL` on each table. `SafeRemoveField` with `MOVE_TO_PENDING` is a no-op in the DB, so the columns stay in place and old pods that still write `scopes` keep working during the rollout. - Prod check from the #127282 thread: 0 rows with `scopes <> 0 AND cardinality(scope_list) = 0` on `sentry_apitoken`, `sentry_apikey`, `sentry_apiauthorization`, `sentry_apigrant`, `sentry_sentryapp`. Replica tables never get `scopes` set; replication only copies `scope_list`. **Phase 2 (separate PR, after this deploys):** `SafeRemoveField(..., deletion_action=DeletionAction.DELETE)` for the same 7 models to actually drop the columns. **Verification** - `makemigrations sentry hybridcloud --check` reports no changes; `sentry upgrade` applies both migrations. - Ran these suites on a fresh test DB: models (apitoken/apigrant/apikey/apiapplication), api token/authorization endpoints, authentication, oauth authorize/token/device, hybridcloud replica, backup exhaustive/imports/exports, sentry app creator/updater, apiapplication deletions. 575 pass. Failures, none touching `scopes`: 3 `test_api_token_details.py::*::test_invalid_user_id` (superuser/elevated-mode path) and 1 backup test that needs Snuba, which isn't available in my sandbox. - prek (ruff, mypy) passes on the changed files. <!-- junior-request-attribution:start --> via **shaun.kaasten**. <!-- junior-request-attribution:end --> <!-- junior-session-footer:start --> <!-- junior-conversation-id:slack%3AD0BAS2BU2TC%3A1791913665.641439 --> -- [View Junior Session](https://junior-prod.sentry.dev/conversations/slack%3AD0BAS2BU2TC%3A1791913665.641439) [[Sentry]](https://sentry.sentry.io/explore/conversations/slack%3AD0BAS2BU2TC%3A1791913665.641439/?project=4510944073809921) <!-- junior-session-footer:end --> Co-authored-by: sentry-junior[bot] <264270552+sentry-junior[bot]@users.noreply.github.com> Co-authored-by: shaun.kaasten <shaun.kaasten@sentry.io>
Phase 2 of 2 for removing the legacy `scopes` BitField. #127386 (phase 1) removed the field from the models and moved the columns to pending deletion. This PR drops the columns. Refs #50475, ENG-3384. Earlier steps: #127282 (removed the `get_scopes()` fallback), #127386 (phase 1). **Phase 1 is fully deployed.** #127386 shipped as getsentry `910f30be`, and GoCD shows it passed on all backend pipelines (S4S2, DE, US, control, ST) on 2026-10-16. No running pod still writes `scopes`. **What changed** - `src/sentry/migrations/1218_drop_scopes_bitfield_columns.py`: `SafeRemoveField(..., deletion_action=DeletionAction.DELETE)` on `apiauthorization`, `apigrant`, `apikey`, `apitoken`, `sentryapp`. - `src/sentry/hybridcloud/migrations/0045_drop_scopes_bitfield_columns.py`: same for `apikeyreplica` and `apitokenreplica`. - `migrations_lockfile.txt`. Migrations only; no code changes. Rebased on master and renumbered from `1216` to `1218` (now depends on `1217_grouprelease_add_new_release_id`), because master added its own `1216` and `1217`. **SQL** ```sql ALTER TABLE "sentry_apiauthorization" DROP COLUMN "scopes" CASCADE; ALTER TABLE "sentry_apigrant" DROP COLUMN "scopes" CASCADE; ALTER TABLE "sentry_apikey" DROP COLUMN "scopes" CASCADE; ALTER TABLE "sentry_apitoken" DROP COLUMN "scopes" CASCADE; ALTER TABLE "sentry_sentryapp" DROP COLUMN "scopes" CASCADE; ALTER TABLE "hybridcloud_apikeyreplica" DROP COLUMN "scopes" CASCADE; ALTER TABLE "hybridcloud_apitokenreplica" DROP COLUMN "scopes" CASCADE; ``` **Verification** - Before the rebase: `makemigrations sentry hybridcloud --check` reported no changes; `sentry upgrade` applied both migrations locally; 181 targeted tests passed on a fresh test DB (apitoken, apigrant, apikey, apiapplication deletions, sentry app updater, hybridcloud replica, api tokens endpoint, oauth token). - The rebase changed only the migration number, its dependency, and the lockfile. CI re-validates the migration graph. - No raw SQL references to these `scopes` columns in `getsentry/sentry` or `getsentry/getsentry`. <!-- junior-request-attribution:start --> via **shaun.kaasten**. <!-- junior-request-attribution:end --> <!-- junior-session-footer:start --> <!-- junior-conversation-id:slack%3AD0BAS2BU2TC%3A1792158314.758249 --> -- [View Junior Session](https://junior-prod.sentry.dev/conversations/slack%3AD0BAS2BU2TC%3A1792158314.758249) [[Sentry]](https://sentry.sentry.io/explore/conversations/slack%3AD0BAS2BU2TC%3A1792158314.758249/?project=4510944073809921) <!-- junior-session-footer:end --> Co-authored-by: sentry-junior[bot] <264270552+sentry-junior[bot]@users.noreply.github.com> Co-authored-by: Shaun Kaasten <shaun.kaasten@sentry.io>
…127386) Phase 1 of 2 for dropping the legacy `scopes` BitField columns. Follow-up to #127282, which made `get_scopes()` read only `scope_list`. Refs #50475, ENG-3384 **What changed** - `src/sentry/models/apiscopes.py`: removed the `scopes` BitField and `ScopesDict` from `HasApiScopes`, including the `assert` that kept the bitfield keys in sync with `ApiScopes`. New scopes no longer need an `AlterField` migration. - `src/sentry/models/apigrant.py`: removed `ApiGrant`'s own `scopes` BitField. - `src/sentry/migrations/1215_remove_scopes_bitfield_pending.py`: makes `scopes` nullable, then `SafeRemoveField(..., MOVE_TO_PENDING)` on `apiauthorization`, `apigrant`, `apikey`, `apitoken`, `sentryapp`. - `src/sentry/hybridcloud/migrations/0044_remove_scopes_bitfield_pending.py`: same for `apikeyreplica` and `apitokenreplica`. - Tests: removed direct `scopes=` bitfield use from `test_apitoken.py`, `test_apiapplication.py` (deletions), and `test_sentry_app_updater.py`. **Deploy safety** - The only SQL is `ALTER COLUMN "scopes" DROP NOT NULL` on each table. `SafeRemoveField` with `MOVE_TO_PENDING` is a no-op in the DB, so the columns stay in place and old pods that still write `scopes` keep working during the rollout. - Prod check from the #127282 thread: 0 rows with `scopes <> 0 AND cardinality(scope_list) = 0` on `sentry_apitoken`, `sentry_apikey`, `sentry_apiauthorization`, `sentry_apigrant`, `sentry_sentryapp`. Replica tables never get `scopes` set; replication only copies `scope_list`. **Phase 2 (separate PR, after this deploys):** `SafeRemoveField(..., deletion_action=DeletionAction.DELETE)` for the same 7 models to actually drop the columns. **Verification** - `makemigrations sentry hybridcloud --check` reports no changes; `sentry upgrade` applies both migrations. - Ran these suites on a fresh test DB: models (apitoken/apigrant/apikey/apiapplication), api token/authorization endpoints, authentication, oauth authorize/token/device, hybridcloud replica, backup exhaustive/imports/exports, sentry app creator/updater, apiapplication deletions. 575 pass. Failures, none touching `scopes`: 3 `test_api_token_details.py::*::test_invalid_user_id` (superuser/elevated-mode path) and 1 backup test that needs Snuba, which isn't available in my sandbox. - prek (ruff, mypy) passes on the changed files. <!-- junior-request-attribution:start --> via **shaun.kaasten**. <!-- junior-request-attribution:end --> <!-- junior-session-footer:start --> <!-- junior-conversation-id:slack%3AD0BAS2BU2TC%3A1791913665.641439 --> -- [View Junior Session](https://junior-prod.sentry.dev/conversations/slack%3AD0BAS2BU2TC%3A1791913665.641439) [[Sentry]](https://sentry.sentry.io/explore/conversations/slack%3AD0BAS2BU2TC%3A1791913665.641439/?project=4510944073809921) <!-- junior-session-footer:end --> Co-authored-by: sentry-junior[bot] <264270552+sentry-junior[bot]@users.noreply.github.com> Co-authored-by: shaun.kaasten <shaun.kaasten@sentry.io>
Phase 2 of 2 for removing the legacy `scopes` BitField. #127386 (phase 1) removed the field from the models and moved the columns to pending deletion. This PR drops the columns. Refs #50475, ENG-3384. Earlier steps: #127282 (removed the `get_scopes()` fallback), #127386 (phase 1). **Phase 1 is fully deployed.** #127386 shipped as getsentry `910f30be`, and GoCD shows it passed on all backend pipelines (S4S2, DE, US, control, ST) on 2026-10-16. No running pod still writes `scopes`. **What changed** - `src/sentry/migrations/1218_drop_scopes_bitfield_columns.py`: `SafeRemoveField(..., deletion_action=DeletionAction.DELETE)` on `apiauthorization`, `apigrant`, `apikey`, `apitoken`, `sentryapp`. - `src/sentry/hybridcloud/migrations/0045_drop_scopes_bitfield_columns.py`: same for `apikeyreplica` and `apitokenreplica`. - `migrations_lockfile.txt`. Migrations only; no code changes. Rebased on master and renumbered from `1216` to `1218` (now depends on `1217_grouprelease_add_new_release_id`), because master added its own `1216` and `1217`. **SQL** ```sql ALTER TABLE "sentry_apiauthorization" DROP COLUMN "scopes" CASCADE; ALTER TABLE "sentry_apigrant" DROP COLUMN "scopes" CASCADE; ALTER TABLE "sentry_apikey" DROP COLUMN "scopes" CASCADE; ALTER TABLE "sentry_apitoken" DROP COLUMN "scopes" CASCADE; ALTER TABLE "sentry_sentryapp" DROP COLUMN "scopes" CASCADE; ALTER TABLE "hybridcloud_apikeyreplica" DROP COLUMN "scopes" CASCADE; ALTER TABLE "hybridcloud_apitokenreplica" DROP COLUMN "scopes" CASCADE; ``` **Verification** - Before the rebase: `makemigrations sentry hybridcloud --check` reported no changes; `sentry upgrade` applied both migrations locally; 181 targeted tests passed on a fresh test DB (apitoken, apigrant, apikey, apiapplication deletions, sentry app updater, hybridcloud replica, api tokens endpoint, oauth token). - The rebase changed only the migration number, its dependency, and the lockfile. CI re-validates the migration graph. - No raw SQL references to these `scopes` columns in `getsentry/sentry` or `getsentry/getsentry`. <!-- junior-request-attribution:start --> via **shaun.kaasten**. <!-- junior-request-attribution:end --> <!-- junior-session-footer:start --> <!-- junior-conversation-id:slack%3AD0BAS2BU2TC%3A1792158314.758249 --> -- [View Junior Session](https://junior-prod.sentry.dev/conversations/slack%3AD0BAS2BU2TC%3A1792158314.758249) [[Sentry]](https://sentry.sentry.io/explore/conversations/slack%3AD0BAS2BU2TC%3A1792158314.758249/?project=4510944073809921) <!-- junior-session-footer:end --> Co-authored-by: sentry-junior[bot] <264270552+sentry-junior[bot]@users.noreply.github.com> Co-authored-by: Shaun Kaasten <shaun.kaasten@sentry.io>
HasApiScopes.get_scopes()andApiGrant.get_scopes()fall back to the legacyscopesBitField whenscope_listis empty. Nothing writes the BitField anymore. Every write path setsscope_list, and hybrid cloud replication copies onlyscope_list.scope_listwas backfilled from the BitField in 2017 (#5139). This change makesscope_listthe only source of truth for scopes.This is the first step toward removing the
scopescolumn (#50475 / ENG-3384). The field and its migrations are left untouched here, so this PR is code-only and easy to revert.Two tests built tokens only through the BitField and are updated to use
scope_list.Prod data check: on the control DB, no rows have a nonzero
scopeswith an emptyscope_listinsentry_apitoken,sentry_apikey,sentry_apiauthorization,sentry_apigrant, orsentry_sentryapp(0 rows each). Nothing relies on the fallback today. The cell-silo token/key replicas only receivescope_listthrough replication, so they never depend on the BitField.via shaun.kaasten.
--
View Junior Session [Sentry]