Repository navigation
chore(releases): swap new_id into the release primary key - #126200
Conversation
2828809 to
a083963
Compare
|
This PR has a migration; here is the generated SQL for for BEGIN;
--
-- Custom state/database change combination
--
DO $$
DECLARE
id_sequence text;
next_id bigint;
pk_name text;
fk record;
fk_drop text;
fk_add text;
fk_drops text[] := '{}';
fk_adds text[] := '{}';
BEGIN
LOCK TABLE "sentry_release" IN ACCESS EXCLUSIVE MODE;
id_sequence := pg_get_serial_sequence('sentry_release', 'id');
IF id_sequence IS NULL THEN
RAISE EXCEPTION 'sentry_release.id has no sequence to carry over';
END IF;
EXECUTE format(
'SELECT CASE WHEN is_called THEN last_value + 1 ELSE last_value END FROM %s', id_sequence
) INTO next_id;
FOR fk IN
SELECT
conrelid::regclass AS table_name,
conname AS constraint_name,
pg_get_constraintdef(oid) AS definition,
convalidated AS validated
FROM pg_constraint
WHERE contype = 'f' AND confrelid = 'sentry_release'::regclass
LOOP
-- The primary key cannot be dropped while a foreign key still depends on it.
fk_drop := format(
'ALTER TABLE %s DROP CONSTRAINT %I', fk.table_name, fk.constraint_name
);
-- Queued rather than run, so every definition is read before the first drop.
fk_drops := array_append(fk_drops, fk_drop);
-- The same definition verbatim, which binds it to whichever column is named id then.
fk_add := format(
'ALTER TABLE %s ADD CONSTRAINT %I %s',
fk.table_name, fk.constraint_name, fk.definition
);
-- Skips the scan of existing rows; new rows are still checked.
IF fk.validated THEN
fk_add := fk_add || ' NOT VALID';
END IF;
-- Queued for after the column rename, so sentry_release(id) is the wide column.
fk_adds := array_append(fk_adds, fk_add);
END LOOP;
FOREACH fk_drop IN ARRAY fk_drops LOOP
EXECUTE fk_drop;
END LOOP;
-- One of these two is how the old column got its values; the other is a no-op.
ALTER TABLE "sentry_release" ALTER COLUMN "id" DROP IDENTITY IF EXISTS;
ALTER TABLE "sentry_release" ALTER COLUMN "id" DROP DEFAULT;
EXECUTE format('DROP SEQUENCE IF EXISTS %s', id_sequence);
SELECT constraint_name INTO pk_name
FROM information_schema.table_constraints
WHERE table_schema = current_schema()
AND table_name = 'sentry_release'
AND constraint_type = 'PRIMARY KEY';
EXECUTE format('ALTER TABLE "sentry_release" DROP CONSTRAINT %I', pk_name);
-- Naming the constraint renames the prebuilt index onto it; inheriting would leave the
-- key called after the unique index.
ALTER TABLE "sentry_release" ADD CONSTRAINT "sentry_release_pkey"
PRIMARY KEY USING INDEX "sentry_release_new_id_uniq";
DROP INDEX IF EXISTS "sentry_release_id_organization_id";
ALTER INDEX IF EXISTS "sentry_release_new_id_organization_id"
RENAME TO "sentry_release_id_organization_id";
-- Swap column names.
ALTER TABLE "sentry_release" RENAME COLUMN "id" TO "id_tmp";
ALTER TABLE "sentry_release" RENAME COLUMN "new_id" TO "id";
ALTER TABLE "sentry_release" RENAME COLUMN "id_tmp" TO "new_id";
EXECUTE format(
'ALTER TABLE "sentry_release" ALTER COLUMN "id" '
'ADD GENERATED BY DEFAULT AS IDENTITY (START WITH %s)', next_id
);
FOREACH fk_add IN ARRAY fk_adds LOOP
EXECUTE fk_add;
END LOOP;
END $$;
COMMIT; |
| FOR fk IN | ||
| SELECT | ||
| conrelid::regclass AS table_name, | ||
| conname AS constraint_name, | ||
| pg_get_constraintdef(oid) AS definition, | ||
| convalidated AS validated | ||
| FROM pg_constraint | ||
| WHERE contype = 'f' AND confrelid = 'sentry_release'::regclass | ||
| LOOP | ||
| -- The primary key cannot be dropped while a foreign key still depends on it. | ||
| fk_drop := format( | ||
| 'ALTER TABLE %s DROP CONSTRAINT %I', fk.table_name, fk.constraint_name | ||
| ); | ||
| -- Queued rather than run, so every definition is read before the first drop. | ||
| fk_drops := array_append(fk_drops, fk_drop); | ||
|
|
||
| -- The same definition verbatim, which binds it to whichever column is named id then. | ||
| fk_add := format( | ||
| 'ALTER TABLE %s ADD CONSTRAINT %I %s', | ||
| fk.table_name, fk.constraint_name, fk.definition | ||
| ); | ||
| -- Skips the scan of existing rows; new rows are still checked. | ||
| IF fk.validated THEN | ||
| fk_add := fk_add || ' NOT VALID'; | ||
| END IF; | ||
| -- Queued for after the column rename, so sentry_release(id) is the wide column. | ||
| fk_adds := array_append(fk_adds, fk_add); | ||
| END LOOP; |
There was a problem hiding this comment.
This loop goes through every FK that points at sentry_releases and writes a statement to drop them, and one to put them back.
Pseudocode:
fk_drops, fk_adds = [], []
for fk in foreign_keys_referencing("sentry_release"):
fk_drops.append(f"ALTER TABLE {fk.table} DROP CONSTRAINT {fk.name}")
fk_add = f"ALTER TABLE {fk.table} ADD CONSTRAINT {fk.name} {fk.definition}"
if fk.validated:
fk_add += " NOT VALID"
fk_adds.append(fk_add)
We need to drop the FKs before dropping the PK for the swap, and then re-add them once the new PK is in. We have to read the FKs before dropping them
wedamija
left a comment
There was a problem hiding this comment.
Do you know how long this will take to run? sentry_release is a hotter table than most of the other ones we've touched here. I assume it's a pretty fast operation, but I just want to be sure here
| ); | ||
|
|
||
| FOREACH fk_add IN ARRAY fk_adds LOOP | ||
| EXECUTE fk_add; |
There was a problem hiding this comment.
Shouldn't we also enable validation of the foreign keys? It would need to be done outside of the exclusive lock but could be done as a later step in the migration operations.
There was a problem hiding this comment.
the migration is atomic = true though, wouldn't adding validation just lengthen the transaction and cause lock latency?
There was a problem hiding this comment.
I think we can validate separately as well, since I think validating doesn't take an exclusive lock? I have some vague memory of this. Not sure if it should be a separate migration?
We'd have to disable the statement timeout since I think it'd take more than 10s
There was a problem hiding this comment.
I can have it as a followup, but I guess the problem is that we'd know anyway if the foreign keys got messed up (big incident)
There was a problem hiding this comment.
I think we can validate separately as well, since I think validating doesn't take an exclusive lock?
Yes, validating a constraint takes a share lock.
... The validation step does not need to lock out concurrent updates, since it knows that other transactions will be enforcing the constraint for rows that they insert or update; only pre-existing rows need to be checked. Hence, validation acquires only a SHARE UPDATE EXCLUSIVE lock on the table being altered. (If the constraint is a foreign key then a ROW SHARE lock is also required on the table referenced by the constraint.) In addition to improving concurrency, it can be useful to use NOT VALID and VALIDATE CONSTRAINT in cases where the table is known to contain pre-existing violations. Once the constraint is in place, no new violations can be inserted, and the existing problems can be corrected at leisure until VALIDATE CONSTRAINT finally succeeds.
I can have it as a followup, but I guess the problem is that we'd know anyway if the foreign keys got messed up (big incident)
Ok. Newly added rows will be validated as they are inserted updated, and the existing rows should be valid as we are working with data that is currently valid.
I was mostly interested in keeping the time that the constraints were not fully validated as short as possible.
There was a problem hiding this comment.
Ok. Newly added rows will be validated as they are inserted updated, and the existing rows should be valid as we are working with data that is currently valid.
so doing as a followup migration? I can have that lined up
Swap itself is milliseconds at most. We might wait on the lock for several seconds but that would timeout after 10 seconds anyway |
a083963 to
2a63bd6
Compare
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 2a63bd6. Configure here.
|
This PR has a migration; here is the generated SQL for for BEGIN;
--
-- Custom state/database change combination
--
DO $$
DECLARE
id_sequence text;
next_id bigint;
pk_name text;
fk record;
fk_drop text;
fk_add text;
fk_drops text[] := '{}';
fk_adds text[] := '{}';
BEGIN
LOCK TABLE "sentry_release" IN ACCESS EXCLUSIVE MODE;
id_sequence := pg_get_serial_sequence('sentry_release', 'id');
IF id_sequence IS NULL THEN
RAISE EXCEPTION 'sentry_release.id has no sequence to carry over';
END IF;
EXECUTE format(
'SELECT CASE WHEN is_called THEN last_value + 1 ELSE last_value END FROM %s', id_sequence
) INTO next_id;
FOR fk IN
SELECT
conrelid::regclass AS table_name,
conname AS constraint_name,
pg_get_constraintdef(oid) AS definition,
convalidated AS validated
FROM pg_constraint
WHERE contype = 'f' AND confrelid = 'sentry_release'::regclass
LOOP
-- The primary key cannot be dropped while a foreign key still depends on it.
fk_drop := format(
'ALTER TABLE %s DROP CONSTRAINT %I', fk.table_name, fk.constraint_name
);
-- Queued rather than run, so every definition is read before the first drop.
fk_drops := array_append(fk_drops, fk_drop);
-- The same definition verbatim, which binds it to whichever column is named id then.
fk_add := format(
'ALTER TABLE %s ADD CONSTRAINT %I %s',
fk.table_name, fk.constraint_name, fk.definition
);
-- Skips the scan of existing rows; new rows are still checked.
IF fk.validated THEN
fk_add := fk_add || ' NOT VALID';
END IF;
-- Queued for after the column rename, so sentry_release(id) is the wide column.
fk_adds := array_append(fk_adds, fk_add);
END LOOP;
FOREACH fk_drop IN ARRAY fk_drops LOOP
EXECUTE fk_drop;
END LOOP;
-- One of these two is how the old column got its values; the other is a no-op.
ALTER TABLE "sentry_release" ALTER COLUMN "id" DROP IDENTITY IF EXISTS;
ALTER TABLE "sentry_release" ALTER COLUMN "id" DROP DEFAULT;
EXECUTE format('DROP SEQUENCE IF EXISTS %s', id_sequence);
SELECT constraint_name INTO pk_name
FROM information_schema.table_constraints
WHERE table_schema = current_schema()
AND table_name = 'sentry_release'
AND constraint_type = 'PRIMARY KEY';
EXECUTE format('ALTER TABLE "sentry_release" DROP CONSTRAINT %I', pk_name);
-- Naming the constraint renames the prebuilt index onto it; inheriting would leave the
-- key called after the unique index.
ALTER TABLE "sentry_release" ADD CONSTRAINT "sentry_release_pkey"
PRIMARY KEY USING INDEX "sentry_release_new_id_uniq";
DROP INDEX IF EXISTS "sentry_release_id_organization_id";
ALTER INDEX IF EXISTS "sentry_release_new_id_organization_id"
RENAME TO "sentry_release_id_organization_id";
-- Swap column names.
ALTER TABLE "sentry_release" RENAME COLUMN "id" TO "id_tmp";
ALTER TABLE "sentry_release" RENAME COLUMN "new_id" TO "id";
ALTER TABLE "sentry_release" RENAME COLUMN "id_tmp" TO "new_id";
EXECUTE format(
'ALTER TABLE "sentry_release" ALTER COLUMN "id" '
'ADD GENERATED BY DEFAULT AS IDENTITY (START WITH %s)', next_id
);
FOREACH fk_add IN ARRAY fk_adds LOOP
EXECUTE fk_add;
END LOOP;
END $$;
COMMIT; |
869c01a to
756907a
Compare
|
This PR has a migration; here is the generated SQL for for BEGIN;
--
-- Custom state/database change combination
--
DO $$
DECLARE
id_sequence text;
next_id bigint;
pk_name text;
fk record;
fk_drop text;
fk_add text;
fk_drops text[] := '{}';
fk_adds text[] := '{}';
BEGIN
LOCK TABLE "sentry_release" IN ACCESS EXCLUSIVE MODE;
id_sequence := pg_get_serial_sequence('sentry_release', 'id');
IF id_sequence IS NULL THEN
RAISE EXCEPTION 'sentry_release.id has no sequence to carry over';
END IF;
-- Any ALTER SEQUENCE blocks nextval until commit, so reserve_ids cannot claim an id
-- between the snapshot and the drop; the sequence is dropped below regardless.
EXECUTE format('ALTER SEQUENCE %s CACHE 1', id_sequence);
EXECUTE format(
'SELECT CASE WHEN is_called THEN last_value + 1 ELSE last_value END FROM %s', id_sequence
) INTO next_id;
FOR fk IN
SELECT
conrelid::regclass AS table_name,
conname AS constraint_name,
pg_get_constraintdef(oid) AS definition,
convalidated AS validated
FROM pg_constraint
WHERE contype = 'f' AND confrelid = 'sentry_release'::regclass
LOOP
-- The primary key cannot be dropped while a foreign key still depends on it.
fk_drop := format(
'ALTER TABLE %s DROP CONSTRAINT %I', fk.table_name, fk.constraint_name
);
-- Queued rather than run, so every definition is read before the first drop.
fk_drops := array_append(fk_drops, fk_drop);
-- The same definition verbatim, which binds it to whichever column is named id then.
fk_add := format(
'ALTER TABLE %s ADD CONSTRAINT %I %s',
fk.table_name, fk.constraint_name, fk.definition
);
-- Skips the scan of existing rows; new rows are still checked.
IF fk.validated THEN
fk_add := fk_add || ' NOT VALID';
END IF;
-- Queued for after the column rename, so sentry_release(id) is the wide column.
fk_adds := array_append(fk_adds, fk_add);
END LOOP;
FOREACH fk_drop IN ARRAY fk_drops LOOP
EXECUTE fk_drop;
END LOOP;
-- One of these two is how the old column got its values; the other is a no-op.
ALTER TABLE "sentry_release" ALTER COLUMN "id" DROP IDENTITY IF EXISTS;
ALTER TABLE "sentry_release" ALTER COLUMN "id" DROP DEFAULT;
EXECUTE format('DROP SEQUENCE IF EXISTS %s', id_sequence);
SELECT constraint_name INTO pk_name
FROM information_schema.table_constraints
WHERE table_schema = current_schema()
AND table_name = 'sentry_release'
AND constraint_type = 'PRIMARY KEY';
EXECUTE format('ALTER TABLE "sentry_release" DROP CONSTRAINT %I', pk_name);
-- Naming the constraint renames the prebuilt index onto it; inheriting would leave the
-- key called after the unique index.
ALTER TABLE "sentry_release" ADD CONSTRAINT "sentry_release_pkey"
PRIMARY KEY USING INDEX "sentry_release_new_id_uniq";
DROP INDEX IF EXISTS "sentry_release_id_organization_id";
ALTER INDEX IF EXISTS "sentry_release_new_id_organization_id"
RENAME TO "sentry_release_id_organization_id";
-- Swap column names.
ALTER TABLE "sentry_release" RENAME COLUMN "id" TO "id_tmp";
ALTER TABLE "sentry_release" RENAME COLUMN "new_id" TO "id";
ALTER TABLE "sentry_release" RENAME COLUMN "id_tmp" TO "new_id";
EXECUTE format(
'ALTER TABLE "sentry_release" ALTER COLUMN "id" '
'ADD GENERATED BY DEFAULT AS IDENTITY (START WITH %s)', next_id
);
FOREACH fk_add IN ARRAY fk_adds LOOP
EXECUTE fk_add;
END LOOP;
END $$;
COMMIT; |
Unlike the organizationmember_teams swap, sentry_release.id has foreign keys pointing at it, which block the primary key drop and pin the old column by position. The block re-creates them against the new column as NOT VALID so each rebuild stays metadata-only.
A table lock does not cover the sequence, so reserve_ids could still take a value between the next_id snapshot and the sequence drop, and the foreign key drops in that gap can wait on locks for seconds. Any ALTER SEQUENCE blocks nextval until commit, which closes the window.
756907a to
30cb491
Compare
|
This PR has a migration; here is the generated SQL for for BEGIN;
--
-- Custom state/database change combination
--
DO $$
DECLARE
id_sequence text;
next_id bigint;
pk_name text;
fk record;
fk_drop text;
fk_add text;
fk_drops text[] := '{}';
fk_adds text[] := '{}';
BEGIN
LOCK TABLE "sentry_release" IN ACCESS EXCLUSIVE MODE;
id_sequence := pg_get_serial_sequence('sentry_release', 'id');
IF id_sequence IS NULL THEN
RAISE EXCEPTION 'sentry_release.id has no sequence to carry over';
END IF;
-- Any ALTER SEQUENCE blocks nextval until commit, so reserve_ids cannot claim an id
-- between the snapshot and the drop; the sequence is dropped below regardless.
EXECUTE format('ALTER SEQUENCE %s CACHE 1', id_sequence);
EXECUTE format(
'SELECT CASE WHEN is_called THEN last_value + 1 ELSE last_value END FROM %s', id_sequence
) INTO next_id;
FOR fk IN
SELECT
conrelid::regclass AS table_name,
conname AS constraint_name,
pg_get_constraintdef(oid) AS definition,
convalidated AS validated
FROM pg_constraint
WHERE contype = 'f' AND confrelid = 'sentry_release'::regclass
LOOP
-- The primary key cannot be dropped while a foreign key still depends on it.
fk_drop := format(
'ALTER TABLE %s DROP CONSTRAINT %I', fk.table_name, fk.constraint_name
);
-- Queued rather than run, so every definition is read before the first drop.
fk_drops := array_append(fk_drops, fk_drop);
-- The same definition verbatim, which binds it to whichever column is named id then.
fk_add := format(
'ALTER TABLE %s ADD CONSTRAINT %I %s',
fk.table_name, fk.constraint_name, fk.definition
);
-- Skips the scan of existing rows; new rows are still checked.
IF fk.validated THEN
fk_add := fk_add || ' NOT VALID';
END IF;
-- Queued for after the column rename, so sentry_release(id) is the wide column.
fk_adds := array_append(fk_adds, fk_add);
END LOOP;
FOREACH fk_drop IN ARRAY fk_drops LOOP
EXECUTE fk_drop;
END LOOP;
-- One of these two is how the old column got its values; the other is a no-op.
ALTER TABLE "sentry_release" ALTER COLUMN "id" DROP IDENTITY IF EXISTS;
ALTER TABLE "sentry_release" ALTER COLUMN "id" DROP DEFAULT;
EXECUTE format('DROP SEQUENCE IF EXISTS %s', id_sequence);
SELECT constraint_name INTO pk_name
FROM information_schema.table_constraints
WHERE table_schema = current_schema()
AND table_name = 'sentry_release'
AND constraint_type = 'PRIMARY KEY';
EXECUTE format('ALTER TABLE "sentry_release" DROP CONSTRAINT %I', pk_name);
-- Naming the constraint renames the prebuilt index onto it; inheriting would leave the
-- key called after the unique index.
ALTER TABLE "sentry_release" ADD CONSTRAINT "sentry_release_pkey"
PRIMARY KEY USING INDEX "sentry_release_new_id_uniq";
DROP INDEX IF EXISTS "sentry_release_id_organization_id";
ALTER INDEX IF EXISTS "sentry_release_new_id_organization_id"
RENAME TO "sentry_release_id_organization_id";
-- Swap column names.
ALTER TABLE "sentry_release" RENAME COLUMN "id" TO "id_tmp";
ALTER TABLE "sentry_release" RENAME COLUMN "new_id" TO "id";
ALTER TABLE "sentry_release" RENAME COLUMN "id_tmp" TO "new_id";
EXECUTE format(
'ALTER TABLE "sentry_release" ALTER COLUMN "id" '
'ADD GENERATED BY DEFAULT AS IDENTITY (START WITH %s)', next_id
);
FOREACH fk_add IN ARRAY fk_adds LOOP
EXECUTE fk_add;
END LOOP;
END $$;
COMMIT; |
resolves INFRENG-726 Adds a post-deployment migration that validates the foreign keys into `sentry_release`, which the 1207 primary key swap re-added as `NOT VALID`. @markstory this is the follow-up you asked for on #126200. Each table is validated in its own transaction with no statement timeout, and `sentry_groupedmessage` goes last. Validation only takes SHARE UPDATE EXCLUSIVE on the child table and ROW SHARE on `sentry_release`, so reads and writes keep flowing. A rerun after an interrupted run skips the keys that are already validated, so it's safe to cancel and restart.

resolves INFRENG-671
Promotes the new_id shadow column to the primary key of sentry_release, so id is bigint from here on, and records the (id, organization_id) index in Django state.
Nine foreign keys reference sentry_release.id, which the organizationmember_teams swap never had to deal with. They block the primary key drop and pin the old column by position, so the swap drops them and re-adds them against the new column as NOT VALID. That keeps the rebuild metadata-only but leaves existing rows unvalidated; validating them later would be a separate scan per referencing table.