postgres-database-migration

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

PostgreSQL Database Migrations

PostgreSQL数据库迁移

A schema migration that works on an empty dev database can fail, lock, or corrupt data on a production table with millions of rows. This guide covers how to assess risk, test against real data, and execute migrations safely.
在空开发数据库上运行正常的schema迁移,在拥有数百万行的生产表上可能会失败、锁表甚至损坏数据。本指南介绍如何评估风险、基于真实数据测试以及安全执行迁移。

DDL Lock Reference

DDL锁参考

Every schema change acquires a lock. The critical question is: does it block reads and writes, and for how long?
每一次schema变更都会获取锁。核心问题是:它是否会阻塞读写?会阻塞多久?

Fast, Non-Blocking Operations

快速非阻塞操作

These complete in milliseconds regardless of table size. They only hold a brief
AccessExclusiveLock
for the catalog update, not for data rewriting.
OperationLock LevelNotes
ADD COLUMN
(nullable, no default)
AccessExclusiveLock
(brief)
Fast. No table rewrite. Metadata-only change.
ADD COLUMN ... DEFAULT x
(PG 11+)
AccessExclusiveLock
(brief)
Fast. Non-volatile defaults stored in catalog, not backfilled.
DROP COLUMN
AccessExclusiveLock
(brief)
Fast. Column marked invisible; space reclaimed by VACUUM over time.
SET DEFAULT
/
DROP DEFAULT
AccessExclusiveLock
(brief)
Metadata change only. Does not touch existing rows.
CREATE INDEX CONCURRENTLY
ShareUpdateExclusiveLock
Non-blocking. Allows reads and writes during build. Slower than regular index creation.
DROP INDEX CONCURRENTLY
ShareUpdateExclusiveLock
Non-blocking. Waits for queries using the index to finish, then drops. No table-level exclusive lock.
RENAME COLUMN
AccessExclusiveLock
(brief)
Metadata change only. Fast.
RENAME TABLE
AccessExclusiveLock
(brief)
Metadata change only. Fast.
ADD CONSTRAINT ... NOT VALID
ShareUpdateExclusiveLock
Adds constraint for new rows only. Does not scan existing data.
VALIDATE CONSTRAINT
ShareUpdateExclusiveLock
Scans existing rows but allows concurrent reads and writes.
CREATE/DROP TRIGGER
ShareRowExclusiveLock
Brief catalog update.
无论表大小如何,这些操作都能在毫秒级完成。它们仅在更新目录时短暂持有
AccessExclusiveLock
,不会重写数据。
操作锁级别说明
ADD COLUMN
(可空,无默认值)
AccessExclusiveLock
(短暂持有)
速度快。 无需表重写,仅修改元数据。
ADD COLUMN ... DEFAULT x
(PG 11+)
AccessExclusiveLock
(短暂持有)
速度快。 非易失性默认值存储在目录中,无需回填。
DROP COLUMN
AccessExclusiveLock
(短暂持有)
速度快。 列被标记为不可见,空间会随时间被VACUUM回收。
SET DEFAULT
/
DROP DEFAULT
AccessExclusiveLock
(短暂持有)
仅修改元数据,不会触碰现有行。
CREATE INDEX CONCURRENTLY
ShareUpdateExclusiveLock
非阻塞。 构建期间允许读写。比普通索引创建慢。
DROP INDEX CONCURRENTLY
ShareUpdateExclusiveLock
非阻塞。 等待使用该索引的查询完成后再删除,无表级排他锁。
RENAME COLUMN
AccessExclusiveLock
(短暂持有)
仅修改元数据,速度快。
RENAME TABLE
AccessExclusiveLock
(短暂持有)
仅修改元数据,速度快。
ADD CONSTRAINT ... NOT VALID
ShareUpdateExclusiveLock
仅对新行生效的约束,不扫描现有数据。
VALIDATE CONSTRAINT
ShareUpdateExclusiveLock
扫描现有行,但允许并发读写。
CREATE/DROP TRIGGER
ShareRowExclusiveLock
短暂的目录更新。

Slow or Blocking Operations

慢速或阻塞操作

These rewrite the table or scan all rows. On large tables, they can lock out all access for seconds to hours.
OperationLock LevelWhy It's Slow
ADD COLUMN ... DEFAULT x
(volatile, e.g.
now()
,
gen_random_uuid()
)
AccessExclusiveLock
Full table rewrite. Every row gets the computed value.
ALTER COLUMN TYPE
(most type changes)
AccessExclusiveLock
Full table rewrite to convert stored data.
SET NOT NULL
(PG < 12, or without existing CHECK)
AccessExclusiveLock
Full table scan to verify no NULLs. See safe pattern below.
ADD CONSTRAINT ... CHECK/UNIQUE/FK
(validated)
AccessExclusiveLock
or
ShareRowExclusiveLock
Scans all rows to verify, blocks writes.
CREATE INDEX
(without CONCURRENTLY)
ShareLock
Blocks writes for the entire build duration.
CLUSTER
AccessExclusiveLock
Rewrites entire table in index order.
VACUUM FULL
AccessExclusiveLock
Rewrites table to reclaim space.
Key insight:
AccessExclusiveLock
blocks everything — reads and writes. Even if the operation itself is fast (milliseconds), it must wait for all in-flight transactions to finish before acquiring the lock. A long-running query or idle transaction can cause an
ALTER TABLE
to hang and queue up all subsequent queries behind it.
这些操作会重写表或扫描所有行。在大表上,它们可能会阻塞所有访问数秒到数小时不等。
操作锁级别慢速原因
ADD COLUMN ... DEFAULT x
(易失性默认值,例如
now()
、
gen_random_uuid()
)
AccessExclusiveLock
全表重写,每一行都会被赋予计算值。
ALTER COLUMN TYPE
(大多数类型变更)
AccessExclusiveLock
全表重写以转换存储的数据。
SET NOT NULL
(PG < 12,或无现有CHECK约束)
AccessExclusiveLock
全表扫描以验证无NULL值。参见下方的安全模式。
ADD CONSTRAINT ... CHECK/UNIQUE/FK
(需验证)
AccessExclusiveLock
或
ShareRowExclusiveLock
扫描所有行进行验证,阻塞写入。
CREATE INDEX
(不带CONCURRENTLY)
ShareLock
整个构建期间阻塞写入。
CLUSTER
AccessExclusiveLock
按索引顺序重写整个表。
VACUUM FULL
AccessExclusiveLock
重写表以回收空间。
核心要点:
AccessExclusiveLock
会阻塞所有操作——包括读和写。即使操作本身很快(毫秒级),它也必须等待所有进行中的事务完成后才能获取锁。一个长时间运行的查询或空闲事务可能导致
ALTER TABLE
挂起,并让所有后续查询在其后排队等待。

Safe Migration Patterns

安全迁移模式

Add a Column

添加列

sql
-- SAFE: nullable column, no default — instant
ALTER TABLE orders ADD COLUMN tracking_number TEXT;

-- SAFE (PG 11+): column with non-volatile default — instant
ALTER TABLE orders ADD COLUMN priority INTEGER NOT NULL DEFAULT 0;

-- UNSAFE: column with volatile default — full table rewrite
-- DON'T: ALTER TABLE orders ADD COLUMN created_at TIMESTAMPTZ DEFAULT now();
-- DO: add nullable, then backfill, then set default + NOT NULL
ALTER TABLE orders ADD COLUMN created_at TIMESTAMPTZ;
-- Backfill in batches (see Backfill section)
ALTER TABLE orders ALTER COLUMN created_at SET DEFAULT now();
ALTER TABLE orders ALTER COLUMN created_at SET NOT NULL;  -- only if PG12+ or CHECK exists
sql
-- SAFE: nullable column, no default — instant
ALTER TABLE orders ADD COLUMN tracking_number TEXT;

-- SAFE (PG 11+): column with non-volatile default — instant
ALTER TABLE orders ADD COLUMN priority INTEGER NOT NULL DEFAULT 0;

-- UNSAFE: column with volatile default — full table rewrite
-- DON'T: ALTER TABLE orders ADD COLUMN created_at TIMESTAMPTZ DEFAULT now();
-- DO: add nullable, then backfill, then set default + NOT NULL
ALTER TABLE orders ADD COLUMN created_at TIMESTAMPTZ;
-- Backfill in batches (see Backfill section)
ALTER TABLE orders ALTER COLUMN created_at SET DEFAULT now();
ALTER TABLE orders ALTER COLUMN created_at SET NOT NULL;  -- only if PG12+ or CHECK exists

Drop a Column

删除列

sql
-- SAFE: instant (column marked invisible, space reclaimed by VACUUM)
ALTER TABLE orders DROP COLUMN old_status;
Application coordination: Ensure your application no longer references the column before dropping it. For zero-downtime deploys, this requires two steps:
  1. Deploy code that doesn't read/write the column
  2. Then drop the column in a separate migration
Security caveat:
DROP COLUMN
does not physically delete the data. The column is marked as dropped in
pg_attribute
but the values remain on disk until
VACUUM
reclaims the space — and even then, a superuser could recover them. If the column contains sensitive data, run
VACUUM FULL
on the table after dropping, or use dump/restore to ensure the data is truly gone.
sql
-- SAFE: instant (column marked invisible, space reclaimed by VACUUM)
ALTER TABLE orders DROP COLUMN old_status;
应用配合: 在删除列之前,请确保你的应用不再引用该列。对于零停机部署,这需要两个步骤:
  1. 部署不再读写该列的代码
  2. 然后在单独的迁移中删除该列
安全注意事项:
DROP COLUMN
不会物理删除数据。列仅在
pg_attribute
中被标记为已删除,但其值仍保留在磁盘上,直到
VACUUM
回收空间——即便如此,超级用户仍可以恢复数据。如果列包含敏感数据,请在删除后对表运行
VACUUM FULL
,或使用导出/导入的方式确保数据被彻底清除。

Rename a Column

重命名列

sql
-- SAFE: instant metadata change
ALTER TABLE orders RENAME COLUMN status TO order_status;
Warning: This breaks any application code, views, or functions that reference the old column name. For zero-downtime deploys, use the column-swap pattern instead:
  1. Add the new column
  2. Deploy code that writes to both columns
  3. Backfill old rows
  4. Deploy code that reads from the new column
  5. Drop the old column
sql
-- SAFE: instant metadata change
ALTER TABLE orders RENAME COLUMN status TO order_status;
警告: 这会破坏所有引用旧列名的应用代码、视图或函数。对于零停机部署,请改用列交换模式:
  1. 添加新列
  2. 部署同时写入两个列的代码
  3. 回填旧行数据
  4. 部署从新列读取的代码
  5. 删除旧列

Change a Column Type

修改列类型

Most type changes rewrite the entire table. Safe alternatives:
sql
-- UNSAFE: full table rewrite, blocks everything
-- DON'T: ALTER TABLE orders ALTER COLUMN amount TYPE NUMERIC(12,2);

-- SAFE: use a new column + backfill
ALTER TABLE orders ADD COLUMN amount_new NUMERIC(12,2);

-- Backfill in batches (see Backfill section below)
UPDATE orders SET amount_new = amount WHERE id BETWEEN 1 AND 10000;
-- ... continue in batches ...

-- Swap columns
ALTER TABLE orders DROP COLUMN amount;
ALTER TABLE orders RENAME COLUMN amount_new TO amount;
Exception: Some casts don't require a rewrite and are fast:
FromToRewrite?
VARCHAR(n)
→
VARCHAR(m)
where m > n
NoMetadata only
VARCHAR(n)
→
TEXT
NoMetadata only
NUMERIC(p,s)
→
NUMERIC(p2,s)
where p2 > p (same scale)
NoMetadata only
INTEGER
→
BIGINT
YesFull rewrite
TIMESTAMP
→
TIMESTAMPTZ
YesFull rewrite
大多数类型变更会重写整个表。安全的替代方案:
sql
-- UNSAFE: full table rewrite, blocks everything
-- DON'T: ALTER TABLE orders ALTER COLUMN amount TYPE NUMERIC(12,2);

-- SAFE: use a new column + backfill
ALTER TABLE orders ADD COLUMN amount_new NUMERIC(12,2);

-- Backfill in batches (see Backfill section below)
UPDATE orders SET amount_new = amount WHERE id BETWEEN 1 AND 10000;
-- ... continue in batches ...

-- Swap columns
ALTER TABLE orders DROP COLUMN amount;
ALTER TABLE orders RENAME COLUMN amount_new TO amount;
例外情况: 某些类型转换不需要重写,速度很快:
源类型目标类型是否需要重写?
VARCHAR(n)
→
VARCHAR(m)
(m > n)
否仅修改元数据
VARCHAR(n)
→
TEXT
否仅修改元数据
NUMERIC(p,s)
→
NUMERIC(p2,s)
(p2 > p,小数位相同)
否仅修改元数据
INTEGER
→
BIGINT
是全表重写
TIMESTAMP
→
TIMESTAMPTZ
是全表重写

Add a NOT NULL Constraint

添加NOT NULL约束

sql
-- PG 18+: simplified two-step pattern
ALTER TABLE orders ALTER COLUMN order_status SET NOT NULL NOT VALID;
ALTER TABLE orders VALIDATE NOT NULL ON order_status;

-- PG 12–17: fast if a valid CHECK constraint already exists
-- Step 1: add CHECK (non-blocking scan)
ALTER TABLE orders ADD CONSTRAINT orders_status_nn CHECK (order_status IS NOT NULL) NOT VALID;
ALTER TABLE orders VALIDATE CONSTRAINT orders_status_nn;

-- Step 2: add NOT NULL (PG12+ recognizes the CHECK and skips the scan)
ALTER TABLE orders ALTER COLUMN order_status SET NOT NULL;

-- Step 3: drop the now-redundant CHECK
ALTER TABLE orders DROP CONSTRAINT orders_status_nn;

-- PG < 12: SET NOT NULL always scans the full table.
-- Ensure no NULLs exist first, then accept the brief lock.
sql
-- PG 18+: simplified two-step pattern
ALTER TABLE orders ALTER COLUMN order_status SET NOT NULL NOT VALID;
ALTER TABLE orders VALIDATE NOT NULL ON order_status;

-- PG 12–17: fast if a valid CHECK constraint already exists
-- Step 1: add CHECK (non-blocking scan)
ALTER TABLE orders ADD CONSTRAINT orders_status_nn CHECK (order_status IS NOT NULL) NOT VALID;
ALTER TABLE orders VALIDATE CONSTRAINT orders_status_nn;

-- Step 2: add NOT NULL (PG12+ recognizes the CHECK and skips the scan)
ALTER TABLE orders ALTER COLUMN order_status SET NOT NULL;

-- Step 3: drop the now-redundant CHECK
ALTER TABLE orders DROP CONSTRAINT orders_status_nn;

-- PG < 12: SET NOT NULL always scans the full table.
-- Ensure no NULLs exist first, then accept the brief lock.

Add a Foreign Key

添加外键

sql
-- UNSAFE: validates all existing rows while holding a heavy lock
-- DON'T: ALTER TABLE orders ADD CONSTRAINT fk_user FOREIGN KEY (user_id) REFERENCES users(id);

-- SAFE: two-step approach
-- Step 1: add without validation (blocks writes briefly, doesn't scan data)
ALTER TABLE orders ADD CONSTRAINT fk_user
    FOREIGN KEY (user_id) REFERENCES users(id) NOT VALID;

-- Step 2: validate existing rows (allows concurrent reads and writes)
ALTER TABLE orders VALIDATE CONSTRAINT fk_user;
sql
-- UNSAFE: validates all existing rows while holding a heavy lock
-- DON'T: ALTER TABLE orders ADD CONSTRAINT fk_user FOREIGN KEY (user_id) REFERENCES users(id);

-- SAFE: two-step approach
-- Step 1: add without validation (blocks writes briefly, doesn't scan data)
ALTER TABLE orders ADD CONSTRAINT fk_user
    FOREIGN KEY (user_id) REFERENCES users(id) NOT VALID;

-- Step 2: validate existing rows (allows concurrent reads and writes)
ALTER TABLE orders VALIDATE CONSTRAINT fk_user;

Add an Index

添加索引

sql
-- UNSAFE on large tables: blocks all writes for the entire build
-- DON'T: CREATE INDEX idx_orders_user ON orders (user_id);

-- SAFE: concurrent index creation (allows reads and writes)
CREATE INDEX CONCURRENTLY idx_orders_user ON orders (user_id);

-- IMPORTANT: if concurrent index creation fails (crashes, deadlock),
-- it leaves an INVALID index behind. Check and clean up:
SELECT indexrelname, idx_scan
FROM pg_stat_user_indexes
WHERE schemaname = 'public'
  AND indexrelname = 'idx_orders_user';

-- Check for invalid indexes
SELECT indexrelid::regclass AS index_name, indisvalid
FROM pg_index
WHERE NOT indisvalid;

-- Drop and retry if invalid
DROP INDEX CONCURRENTLY idx_orders_user;
CREATE INDEX CONCURRENTLY idx_orders_user ON orders (user_id);
sql
-- UNSAFE on large tables: blocks all writes for the entire build
-- DON'T: CREATE INDEX idx_orders_user ON orders (user_id);

-- SAFE: concurrent index creation (allows reads and writes)
CREATE INDEX CONCURRENTLY idx_orders_user ON orders (user_id);

-- IMPORTANT: if concurrent index creation fails (crashes, deadlock),
-- it leaves an INVALID index behind. Check and clean up:
SELECT indexrelname, idx_scan
FROM pg_stat_user_indexes
WHERE schemaname = 'public'
  AND indexrelname = 'idx_orders_user';

-- Check for invalid indexes
SELECT indexrelid::regclass AS index_name, indisvalid
FROM pg_index
WHERE NOT indisvalid;

-- Drop and retry if invalid
DROP INDEX CONCURRENTLY idx_orders_user;
CREATE INDEX CONCURRENTLY idx_orders_user ON orders (user_id);

Add a Unique Constraint

添加唯一约束

sql
-- A UNIQUE constraint creates an index. Use CONCURRENTLY to avoid blocking:

-- Step 1: create a unique index concurrently
CREATE UNIQUE INDEX CONCURRENTLY idx_orders_tracking_uniq ON orders (tracking_number);

-- Step 2: attach it as a constraint (instant)
ALTER TABLE orders ADD CONSTRAINT orders_tracking_uniq UNIQUE USING INDEX idx_orders_tracking_uniq;
sql
-- A UNIQUE constraint creates an index. Use CONCURRENTLY to avoid blocking:

-- Step 1: create a unique index concurrently
CREATE UNIQUE INDEX CONCURRENTLY idx_orders_tracking_uniq ON orders (tracking_number);

-- Step 2: attach it as a constraint (instant)
ALTER TABLE orders ADD CONSTRAINT orders_tracking_uniq UNIQUE USING INDEX idx_orders_tracking_uniq;

Redefine a Primary Key

重新定义主键

Redefining a PK (e.g., switching from
id
to a composite key, or from
int
to
bigint
) requires both a UNIQUE constraint and NOT NULL — both of which can cause long-lasting locks if done naively. The zero-downtime approach builds each ingredient separately:
sql
-- Step 1: add CHECK NOT NULL constraint without validation (brief lock)
ALTER TABLE orders ADD CONSTRAINT orders_new_id_nn
    CHECK (new_id IS NOT NULL) NOT VALID;

-- Step 2: validate existing rows (allows concurrent reads and writes)
ALTER TABLE orders VALIDATE CONSTRAINT orders_new_id_nn;

-- Step 3: build unique index concurrently (non-blocking)
CREATE UNIQUE INDEX CONCURRENTLY idx_orders_new_pkey
    ON orders (new_id);

-- Step 4: drop the old PK
ALTER TABLE orders DROP CONSTRAINT orders_pkey;

-- Step 5: add new PK using the existing index (instant — also implicitly adds NOT NULL)
ALTER TABLE orders ADD CONSTRAINT orders_pkey
    PRIMARY KEY USING INDEX idx_orders_new_pkey;

-- Step 6: drop the now-redundant CHECK constraint
ALTER TABLE orders DROP CONSTRAINT orders_new_id_nn;
Why this works: Step 5 is fast because Postgres reuses the already-built unique index and recognizes the existing CHECK constraint, skipping both the index build and the full-table NOT NULL scan (PG12+).
重新定义主键(例如从
id
切换到复合键,或从
int
切换到
bigint
)需要同时有唯一约束和NOT NULL——如果操作不当,两者都可能导致长时间锁表。零停机方案是分别构建每个组件:
sql
-- Step 1: add CHECK NOT NULL constraint without validation (brief lock)
ALTER TABLE orders ADD CONSTRAINT orders_new_id_nn
    CHECK (new_id IS NOT NULL) NOT VALID;

-- Step 2: validate existing rows (allows concurrent reads and writes)
ALTER TABLE orders VALIDATE CONSTRAINT orders_new_id_nn;

-- Step 3: build unique index concurrently (non-blocking)
CREATE UNIQUE INDEX CONCURRENTLY idx_orders_new_pkey
    ON orders (new_id);

-- Step 4: drop the old PK
ALTER TABLE orders DROP CONSTRAINT orders_pkey;

-- Step 5: add new PK using the existing index (instant — also implicitly adds NOT NULL)
ALTER TABLE orders ADD CONSTRAINT orders_pkey
    PRIMARY KEY USING INDEX idx_orders_new_pkey;

-- Step 6: drop the now-redundant CHECK constraint
ALTER TABLE orders DROP CONSTRAINT orders_new_id_nn;
原理: 第5步速度很快,因为Postgres会复用已构建好的唯一索引,并且识别出现有的CHECK约束,从而跳过索引构建和全表NOT NULL扫描(PG12+)。

Drop a Constraint

删除约束

sql
-- SAFE: instant metadata change
ALTER TABLE orders DROP CONSTRAINT orders_tracking_uniq;

-- If dropping a FK that has a supporting index you no longer need:
ALTER TABLE orders DROP CONSTRAINT fk_user;
DROP INDEX idx_orders_user_id;  -- only if no other queries use it
sql
-- SAFE: instant metadata change
ALTER TABLE orders DROP CONSTRAINT orders_tracking_uniq;

-- If dropping a FK that has a supporting index you no longer need:
ALTER TABLE orders DROP CONSTRAINT fk_user;
DROP INDEX idx_orders_user_id;  -- only if no other queries use it

Backfill Strategies

回填策略

Always backfill in batches — never in a single UPDATE. See backfill-strategies for batch-by-PK patterns, resumable progress tracking, and tuning guidance.
始终分批回填——绝不要用单条UPDATE完成。请参阅 backfill-strategies 了解按主键分批的模式、可恢复的进度跟踪以及调优指南。

Migration Validation

迁移验证

Run validation queries before and after every migration. See validation-queries for the full set of checks: NULL detection, duplicate detection, orphan rows, cast failures, duration estimation, schema verification, data integrity, and query performance.
在每次迁移前后运行验证查询。请参阅 validation-queries 获取完整的检查项:NULL检测、重复检测、孤立行、类型转换失败、耗时估算、schema验证、数据完整性以及查询性能。

Rollback Planning

回滚规划

Every migration should have a rollback plan documented before execution.
每次迁移在执行前都应记录回滚计划。

Reversible Operations

可回滚操作

OperationRollback
ADD COLUMN
DROP COLUMN
ADD CONSTRAINT
DROP CONSTRAINT
CREATE INDEX
DROP INDEX
RENAME COLUMN x TO y
RENAME COLUMN y TO x
SET DEFAULT x
SET DEFAULT old_value
or
DROP DEFAULT
ADD COLUMN new + DROP COLUMN old
Cannot directly undo — need to re-add old column and backfill from a backup
操作回滚方式
ADD COLUMN
DROP COLUMN
ADD CONSTRAINT
DROP CONSTRAINT
CREATE INDEX
DROP INDEX
RENAME COLUMN x TO y
RENAME COLUMN y TO x
SET DEFAULT x
SET DEFAULT old_value
或
DROP DEFAULT
ADD COLUMN new + DROP COLUMN old
无法直接撤销——需要重新添加旧列并从备份回填

Irreversible Operations

不可逆操作

These require restoring from a backup or the database fork to undo:
  • DROP COLUMN
    — data is gone once VACUUM reclaims it
  • ALTER COLUMN TYPE
    with lossy cast (e.g.,
    NUMERIC
    →
    INTEGER
    ,
    TEXT
    →
    VARCHAR(50)
    )
  • DELETE
    /
    TRUNCATE
    during data cleanup
  • DROP TABLE
This is where a database fork is invaluable. If you forked before the migration, the original database has the pre-migration state. If the migration went wrong, your production data is untouched — just delete the fork and start over.
这些操作需要从备份或数据库fork恢复才能撤销:
  • DROP COLUMN
    —— 一旦VACUUM回收空间,数据就会丢失
  • 带有损转换的
    ALTER COLUMN TYPE
    (例如
    NUMERIC
    →
    INTEGER
    、
    TEXT
    →
    VARCHAR(50)
    )
  • 数据清理期间的
    DELETE
    /
    TRUNCATE
  • DROP TABLE
这就是数据库fork的价值所在。 如果你在迁移前fork了数据库,原始数据库仍保留迁移前的状态。如果迁移出错,你的生产数据不会受到任何影响——只需删除fork重新开始即可。

Transaction Strategy

事务策略

There are two approaches for executing multiple DDL statements. Each has tradeoffs:
Wrapped in one transaction — all changes succeed or all roll back. Use this when atomicity matters more than lock duration, and all operations are fast (milliseconds).
sql
BEGIN;

ALTER TABLE orders ADD COLUMN priority INTEGER NOT NULL DEFAULT 0;
ALTER TABLE orders ADD COLUMN tags TEXT[] NOT NULL DEFAULT '{}';
CREATE INDEX ON orders USING GIN (tags);
ALTER TABLE orders DROP COLUMN old_priority;

-- Verify before committing
SELECT column_name, data_type
FROM information_schema.columns
WHERE table_name = 'orders'
ORDER BY ordinal_position;

COMMIT;
-- Or ROLLBACK; if something looks wrong
Separate transactions — each DDL runs and commits independently. Use this when lock duration matters more than atomicity. In a single transaction, all locks are held until
COMMIT
— so if you have 5 DDL statements, the
AccessExclusiveLock
from the first one blocks traffic for the entire duration of all 5. Separate transactions release locks between statements.
sql
-- Each statement auto-commits
ALTER TABLE orders ADD COLUMN priority INTEGER NOT NULL DEFAULT 0;
ALTER TABLE orders ADD COLUMN tags TEXT[] NOT NULL DEFAULT '{}';
ALTER TABLE orders DROP COLUMN old_priority;
The tradeoff: separate transactions can leave the schema in a partially migrated state if a later statement fails. You'll need a rollback plan for each step individually.
Cannot use transactions with:
  • CREATE INDEX CONCURRENTLY
    (explicitly disallowed inside a transaction)
  • DROP INDEX CONCURRENTLY
  • Any statement that requires its own transaction context
执行多条DDL语句有两种方法,各有优劣:
包裹在单个事务中 —— 所有变更要么全部成功,要么全部回滚。当原子性比锁持续时间更重要,且所有操作都很快(毫秒级)时使用此方式。
sql
BEGIN;

ALTER TABLE orders ADD COLUMN priority INTEGER NOT NULL DEFAULT 0;
ALTER TABLE orders ADD COLUMN tags TEXT[] NOT NULL DEFAULT '{}';
CREATE INDEX ON orders USING GIN (tags);
ALTER TABLE orders DROP COLUMN old_priority;

-- Verify before committing
SELECT column_name, data_type
FROM information_schema.columns
WHERE table_name = 'orders'
ORDER BY ordinal_position;

COMMIT;
-- Or ROLLBACK; if something looks wrong
独立事务 —— 每个DDL独立运行并提交。当锁持续时间比原子性更重要时使用此方式。在单个事务中,所有锁都会一直持有到
COMMIT
——所以如果你有5条DDL语句,第一条语句的
AccessExclusiveLock
会在所有5条语句的整个执行期间阻塞流量。独立事务会在语句之间释放锁。
sql
-- Each statement auto-commits
ALTER TABLE orders ADD COLUMN priority INTEGER NOT NULL DEFAULT 0;
ALTER TABLE orders ADD COLUMN tags TEXT[] NOT NULL DEFAULT '{}';
ALTER TABLE orders DROP COLUMN old_priority;
权衡之处: 如果后续语句失败,独立事务可能会让schema处于部分迁移的状态。你需要为每个步骤单独制定回滚计划。
以下操作不能在事务中使用:
  • CREATE INDEX CONCURRENTLY
    (明确禁止在事务内运行)
  • DROP INDEX CONCURRENTLY
  • 任何需要自身事务上下文的语句

Dealing with Long-Running Queries

处理长时间运行的查询

A fast
ALTER TABLE
can still hang if it's waiting to acquire
AccessExclusiveLock
behind a long-running query. Worse, the waiting DDL blocks all subsequent queries too — even simple SELECTs pile up behind it:
Session 1: SELECT COUNT(*) FROM orders;          -- long query, holds AccessShareLock
Session 2: ALTER TABLE orders ADD COLUMN ...;     -- waits for Session 1 (needs AccessExclusiveLock)
Session 3: SELECT * FROM orders WHERE id = 123;   -- BLOCKED by Session 2's lock queue entry
Session 4: INSERT INTO orders (...) VALUES (...);  -- also BLOCKED
-- All sessions freeze until Session 1 finishes and Session 2 completes or times out
This is why
lock_timeout
is critical — without it, a single slow query can cascade into an application-wide outage.
即使是快速的
ALTER TABLE
,如果它在一个长时间运行的查询后等待获取
AccessExclusiveLock
,也可能挂起。更糟糕的是,等待中的DDL会阻塞所有后续查询——哪怕是简单的SELECT也会在其后堆积:
Session 1: SELECT COUNT(*) FROM orders;          -- long query, holds AccessShareLock
Session 2: ALTER TABLE orders ADD COLUMN ...;     -- waits for Session 1 (needs AccessExclusiveLock)
Session 3: SELECT * FROM orders WHERE id = 123;   -- BLOCKED by Session 2's lock queue entry
Session 4: INSERT INTO orders (...) VALUES (...);  -- also BLOCKED
-- All sessions freeze until Session 1 finishes and Session 2 completes or times out
这就是为什么
lock_timeout
至关重要——没有它,一个慢查询可能会级联导致整个应用中断。

Set Timeouts

设置超时

lock_timeout
— How long to wait for a lock before giving up. Use this on every production DDL statement. Without it, an
ALTER TABLE
can queue behind a long-running query and block all subsequent queries behind it indefinitely.
statement_timeout
— How long the statement can run once it has the lock. This is a safety net against unexpectedly slow operations (e.g., a type change that triggers a table rewrite you didn't anticipate). The tradeoff: if the timeout fires mid-operation, the entire statement rolls back — which is safe for DDL (no partial changes), but means a long
CREATE INDEX CONCURRENTLY
could be killed near completion. For that reason, avoid setting
statement_timeout
on operations you know will be slow (like concurrent index builds on large tables) and instead monitor them manually.
Choosing timeout values:
There are two schools of thought:
  • Conservative (50-100ms lock_timeout, hundreds of retries): Minimizes the window where a waiting DDL blocks other queries. Each attempt is nearly invisible to application traffic, but requires retry logic. Best for high-traffic OLTP systems where even a few seconds of blocked queries is unacceptable.
  • Pragmatic (3-5s lock_timeout, few retries): Gives the lock a reasonable chance to be acquired on each attempt, reducing the need for complex retry logic. Acceptable for most applications where brief pauses are tolerable.
Pick based on your traffic profile: the higher your query throughput, the shorter your
lock_timeout
should be — because even a brief queue-up affects more queries per second. For
statement_timeout
, set it to a generous multiple of what you expect the operation to take (e.g., 30s for metadata-only changes, minutes for VALIDATE CONSTRAINT on large tables, disabled for CREATE INDEX CONCURRENTLY).
sql
-- Fail fast instead of blocking all queries behind you
SET lock_timeout = '5s';
SET statement_timeout = '30s';

ALTER TABLE orders ADD COLUMN tracking_number TEXT;

-- If it fails with "canceling statement due to lock timeout":
-- 1. Find what's blocking
SELECT pid, state, query, now() - query_start AS duration
FROM pg_stat_activity
WHERE state != 'idle'
ORDER BY duration DESC;

-- 2. Wait for the blocker to finish, or cancel it if appropriate
-- SELECT pg_cancel_backend(<pid>);

-- 3. Retry the ALTER TABLE
SET lock_timeout = '5s';
ALTER TABLE orders ADD COLUMN tracking_number TEXT;

-- Reset timeouts when done
RESET lock_timeout;
RESET statement_timeout;
lock_timeout
—— 放弃前等待锁的时长。在所有生产环境DDL语句中都应使用此参数。如果没有它,
ALTER TABLE
可能会在一个长时间运行的查询后排队,并无限期阻塞其后的所有查询。
statement_timeout
—— 语句获取锁后可运行的最长时间。这是应对意外慢速操作的安全网(例如,触发了你未预料到的表重写的类型修改)。权衡之处在于:如果超时在操作过程中触发,整个语句会回滚——这对DDL来说是安全的(不会有部分变更),但意味着一个长时间运行的
CREATE INDEX CONCURRENTLY
可能在接近完成时被终止。因此,对于你知道会很慢的操作(比如大表上的并发索引构建),请避免设置
statement_timeout
,而是手动监控它们。
如何选择超时值:
有两种主流思路:
  • 保守策略(50-100ms lock_timeout,数百次重试): 最小化等待中的DDL阻塞其他查询的时间窗口。每次尝试对应用流量几乎不可见,但需要重试逻辑。最适合高流量OLTP系统,这类系统中即使几秒的查询阻塞也是不可接受的。
  • 实用策略(3-5s lock_timeout,少量重试): 给每次尝试合理的锁获取机会,减少对复杂重试逻辑的需求。对于可以容忍短暂停顿的大多数应用来说是可接受的。
根据你的流量情况选择:查询吞吐量越高,
lock_timeout
应该越短——因为即使是短暂的排队,每秒影响的查询数也会更多。对于
statement_timeout
,将其设置为你预期操作耗时的较大倍数(例如,元数据-only操作设为30秒,大表上的VALIDATE CONSTRAINT设为数分钟,CREATE INDEX CONCURRENTLY则禁用)。
sql
-- Fail fast instead of blocking all queries behind you
SET lock_timeout = '5s';
SET statement_timeout = '30s';

ALTER TABLE orders ADD COLUMN tracking_number TEXT;

-- If it fails with "canceling statement due to lock timeout":
-- 1. Find what's blocking
SELECT pid, state, query, now() - query_start AS duration
FROM pg_stat_activity
WHERE state != 'idle'
ORDER BY duration DESC;

-- 2. Wait for the blocker to finish, or cancel it if appropriate
-- SELECT pg_cancel_backend(<pid>);

-- 3. Retry the ALTER TABLE
SET lock_timeout = '5s';
ALTER TABLE orders ADD COLUMN tracking_number TEXT;

-- Reset timeouts when done
RESET lock_timeout;
RESET statement_timeout;

The Retry-With-Timeout Pattern

带超时的重试模式

For automated migration runners, wrap DDL in a retry loop with a short lock timeout:
sql
DO $$
DECLARE
    max_attempts INTEGER := 5;
    attempt INTEGER := 1;
    success BOOLEAN := FALSE;
BEGIN
    WHILE attempt <= max_attempts AND NOT success LOOP
        BEGIN
            SET lock_timeout = '3s';
            -- Replace with your DDL statement
            ALTER TABLE orders ADD COLUMN tracking_number TEXT;
            success := TRUE;
            RAISE NOTICE 'DDL succeeded on attempt %', attempt;
        EXCEPTION
            WHEN lock_not_available THEN
                RAISE NOTICE 'Attempt % failed (lock not available), retrying...', attempt;
                PERFORM pg_sleep(2 * attempt);  -- linear backoff
                attempt := attempt + 1;
        END;
    END LOOP;

    IF NOT success THEN
        RAISE EXCEPTION 'DDL failed after % attempts', max_attempts;
    END IF;
END $$;
This prevents the migration from creating a pile-up of blocked queries behind it. Each attempt either succeeds quickly or gives up and lets normal traffic flow.
Alternative:
NOWAIT
— For the highest-traffic systems, use
LOCK TABLE ... NOWAIT
to test lock availability before running DDL. Unlike
lock_timeout
,
NOWAIT
fails instantly without ever entering the lock queue, so there is zero risk of cascading blocked queries. The tradeoff is more retries:
sql
BEGIN;
LOCK TABLE orders IN ACCESS EXCLUSIVE MODE NOWAIT;
-- If we get here, we have the lock — run DDL
ALTER TABLE orders ADD COLUMN tracking_number TEXT;
COMMIT;
-- If LOCK fails with "could not obtain lock", retry after a short sleep
对于自动化迁移执行器,可以将DDL包裹在带有短锁超时的重试循环中:
sql
DO $$
DECLARE
    max_attempts INTEGER := 5;
    attempt INTEGER := 1;
    success BOOLEAN := FALSE;
BEGIN
    WHILE attempt <= max_attempts AND NOT success LOOP
        BEGIN
            SET lock_timeout = '3s';
            -- Replace with your DDL statement
            ALTER TABLE orders ADD COLUMN tracking_number TEXT;
            success := TRUE;
            RAISE NOTICE 'DDL succeeded on attempt %', attempt;
        EXCEPTION
            WHEN lock_not_available THEN
                RAISE NOTICE 'Attempt % failed (lock not available), retrying...', attempt;
                PERFORM pg_sleep(2 * attempt);  -- linear backoff
                attempt := attempt + 1;
        END;
    END LOOP;

    IF NOT success THEN
        RAISE EXCEPTION 'DDL failed after % attempts', max_attempts;
    END IF;
END $$;
这可以防止迁移在其后方造成大量阻塞查询。每次尝试要么快速成功,要么放弃并让正常流量继续通行。
替代方案:
NOWAIT
—— 对于最高流量的系统,在运行DDL前使用
LOCK TABLE ... NOWAIT
测试锁的可用性。与
lock_timeout
不同,
NOWAIT
会立即失败,不会进入锁队列,因此完全没有级联阻塞查询的风险。权衡之处是需要更多次重试:
sql
BEGIN;
LOCK TABLE orders IN ACCESS EXCLUSIVE MODE NOWAIT;
-- If we get here, we have the lock — run DDL
ALTER TABLE orders ADD COLUMN tracking_number TEXT;
COMMIT;
-- If LOCK fails with "could not obtain lock", retry after a short sleep

Fork-Based Migration Testing

基于Fork的迁移测试

The safest way to test a migration is to run it against a copy of your actual database — same schema, same data, same edge cases. Providers such as Neon support fast database forking. Without database forking, you need to manually dump and restore your database, which can take a long time for large datasets.
测试迁移最安全的方式是在真实数据库的副本上运行——相同的schema、相同的数据、相同的边界情况。Neon 等服务商支持快速数据库fork。如果没有数据库fork功能,你需要手动导出并恢复数据库,对于大数据集这可能会花费很长时间。

With Forking

使用Fork的测试方法

  1. Fork your database — create a full copy using your provider's fork feature (takes seconds)
  2. Inspect the current schema on the fork to confirm it matches production
  3. Run your migration on the fork
  4. Validate — run your checks (see Pre/Post-Migration Validation sections above)
  5. If it worked: apply the same migration to production
  6. If it failed: delete the fork — your production database is untouched
This catches problems that never show up in empty test databases:
  • Data that violates a new constraint
  • Type casts that fail on real values
  • Migrations that are fast on 100 rows but lock the table for minutes on 10 million
  • Index creation that runs out of memory or disk space
Limitation: fork-based testing runs your migration in isolation — it won't catch issues caused by concurrent database traffic (e.g., lock contention under load, deadlocks with concurrent writes, or replication lag from heavy WAL generation). For most applications, fork-based testing is sufficient. For very high-uptime applications, use PgDog's mirroring feature to replay production traffic against the fork — it reproduces queries byte-for-byte with realistic timing, and you can filter to DDL-only or DML-only and control exposure percentage to ramp up gradually.
  1. Fork你的数据库 —— 使用服务商的fork功能创建完整副本(仅需几秒)
  2. 检查fork上的当前schema,确认与生产环境一致
  3. 在fork上运行你的迁移
  4. 验证 —— 运行你的检查项(参见上文的迁移前后验证部分)
  5. 如果成功: 将相同的迁移应用到生产环境
  6. 如果失败: 删除fork——你的生产数据库不会受到任何影响
这可以发现那些在空测试数据库上永远不会出现的问题:
  • 违反新约束的数据
  • 在真实值上失败的类型转换
  • 在100行上运行很快但在1000万行上锁表数分钟的迁移
  • 耗尽内存或磁盘空间的索引创建
局限性: 基于fork的测试是在隔离环境中运行迁移——它无法发现由并发数据库流量导致的问题(例如,负载下的锁争用、并发写入导致的死锁,或大量WAL生成导致的复制延迟)。对于大多数应用,基于fork的测试已经足够。对于要求极高可用性的应用,可以使用 PgDog 的镜像功能将生产流量重放到fork上——它会逐字节复现查询并保留真实的时间特性,你还可以过滤为仅DDL或仅DML,并控制流量百分比逐步放量。

Without Forking

无Fork时的测试方案

Create a test database from a backup or dump:
bash
undefined
从备份或转储创建测试数据库:
bash
undefined

Dump your production database

Dump your production database

pg_dump -Fc my_app_db > backup.dump
pg_dump -Fc my_app_db > backup.dump

Restore into a test database

Restore into a test database

createdb migration_test pg_restore -d migration_test backup.dump
createdb migration_test pg_restore -d migration_test backup.dump

Or clone from a live database (requires downtime on source during copy)

Or clone from a live database (requires downtime on source during copy)

createdb migration_test -T my_app_db
undefined
createdb migration_test -T my_app_db
undefined

Complete Migration Example

完整迁移示例

For a full end-to-end walkthrough (plan, fork, run, validate, apply, clean up), see complete-example.
有关完整的端到端演练(规划、fork、运行、验证、应用、清理),请参阅 complete-example。

Advanced Considerations

高级注意事项

Subtransactions in PL/pgSQL retry loops: The
BEGIN/EXCEPTION WHEN/END
block in the retry-with-timeout pattern creates implicit subtransactions. Under high write throughput, this can trigger SubtransSLRU contention on replicas — especially if the retry loop runs as a long-lived transaction with many attempts. If you see replica lag during retries, move the retry logic to the application layer (separate transactions per attempt) instead of using PL/pgSQL exception handling.
Autovacuum can block VALIDATE CONSTRAINT:
VALIDATE CONSTRAINT
acquires
ShareUpdateExclusiveLock
, which conflicts with autovacuum running in transaction ID wraparound prevention mode. If
VALIDATE
hangs unexpectedly, check
pg_stat_activity
for autovacuum processes on the same table. You may need to wait for wraparound-prevention autovacuum to finish — do not cancel it, as that can lead to data loss if the table approaches the XID wraparound limit.
PL/pgSQL重试循环中的子事务: 带超时重试模式中的
BEGIN/EXCEPTION WHEN/END
块会创建隐式子事务。在高写入吞吐量下,这可能会在副本上触发SubtransSLRU争用——尤其是当重试循环作为一个长生命周期事务运行且有多次尝试时。如果你在重试期间看到副本延迟,请将重试逻辑移到应用层(每次尝试使用独立事务),而不是使用PL/pgSQL异常处理。
Autovacuum可能阻塞VALIDATE CONSTRAINT:
VALIDATE CONSTRAINT
会获取
ShareUpdateExclusiveLock
,这与运行在事务ID回卷预防模式下的autovacuum冲突。如果
VALIDATE
意外挂起,请检查
pg_stat_activity
中同一张表上的autovacuum进程。你可能需要等待回卷预防的autovacuum完成——不要取消它,因为如果表接近XID回卷限制,可能会导致数据丢失。

Common Pitfalls

常见陷阱

  1. Testing migrations on empty tables — a migration that runs in 1ms on an empty table can lock a 10M-row table for minutes. Always test against realistic data volumes.
  2. Forgetting
    CONCURRENTLY
    on index creation
    —
    CREATE INDEX
    (without
    CONCURRENTLY
    ) blocks all writes. On a table with active traffic, this causes downtime.
  3. Adding NOT NULL without the two-step pattern — on large tables in PG < 12,
    SET NOT NULL
    scans every row while holding
    AccessExclusiveLock
    . Use the CHECK constraint pattern.
  4. No lock timeout — a fast ALTER TABLE can block behind a long-running query, and every subsequent query stacks up behind it. Always
    SET lock_timeout
    for production DDL.
  5. Backfilling in one big transaction — a single
    UPDATE orders SET x = y
    on 10M rows generates enormous WAL, bloats the table, and holds locks for the entire duration. Always batch.
  6. Leaving invalid indexes behind — if
    CREATE INDEX CONCURRENTLY
    fails, it leaves an invisible invalid index that consumes space and slows writes. Check
    pg_index.indisvalid
    after every concurrent index operation.
  7. Dropping columns before updating application code — in a running system, the old code still references the column. Deploy the code change first, then drop the column in a subsequent migration.
  8. Not checking replication lag — large backfills generate heavy WAL. If you have read replicas, monitor
    pg_stat_replication
    during and after the migration.
  9. Assuming ALTER COLUMN TYPE is safe — most type changes rewrite the entire table. Use the add-new-column + backfill + swap pattern for large tables.
  1. 在空表上测试迁移 —— 在空表上1ms运行完的迁移,在1000万行的表上可能锁表数分钟。始终使用真实数据量进行测试。
  2. 创建索引时忘记加
    CONCURRENTLY
    —— (不带
    CONCURRENTLY
    的)
    CREATE INDEX
    会阻塞所有写入。在有活跃流量的表上,这会导致停机。
  3. 不使用两步模式添加NOT NULL —— 在PG < 12的大表上,
    SET NOT NULL
    会在持有
    AccessExclusiveLock
    的同时扫描每一行。请使用CHECK约束模式。
  4. 没有设置锁超时 —— 快速的ALTER TABLE可能在长时间运行的查询后阻塞,所有后续查询都会在其后堆积。生产环境DDL务必设置
    lock_timeout
    。
  5. 在一个大事务中回填 —— 在1000万行上运行单条
    UPDATE orders SET x = y
    会生成巨大的WAL,导致表膨胀,并且在整个期间持有锁。始终分批操作。
  6. 遗留无效索引 —— 如果
    CREATE INDEX CONCURRENTLY
    失败,会留下一个不可见的无效索引,占用空间并拖慢写入。每次并发索引操作后都要检查
    pg_index.indisvalid
    。
  7. 在更新应用代码前删除列 —— 在运行中的系统里,旧代码仍在引用该列。请先部署代码变更,再在后续迁移中删除列。
  8. 不检查复制延迟 —— 大量回填会生成很重的WAL。如果你有读副本,请在迁移期间和之后监控
    pg_stat_replication
    。
  9. 假设ALTER COLUMN TYPE是安全的 —— 大多数类型变更会重写整个表。对于大表,请使用“添加新列+回填+交换”的模式。