postgres-database-migration
Compare original and translation side by side
🇺🇸
Original
English🇨🇳
Translation
ChinesePostgreSQL Database Migrations
PostgreSQL数据库迁移
A schema migration that works on an empty dev database can fail, lock, or corrupt data on a production table with millions of rows. This guide covers how to assess risk, test against real data, and execute migrations safely.
在空开发数据库上运行正常的schema迁移,在拥有数百万行的生产表上可能会失败、锁表甚至损坏数据。本指南介绍如何评估风险、基于真实数据测试以及安全执行迁移。
DDL Lock Reference
DDL锁参考
Every schema change acquires a lock. The critical question is: does it block reads and writes, and for how long?
每一次schema变更都会获取锁。核心问题是:它是否会阻塞读写?会阻塞多久?
Fast, Non-Blocking Operations
快速非阻塞操作
These complete in milliseconds regardless of table size. They only hold a brief for the catalog update, not for data rewriting.
AccessExclusiveLock| Operation | Lock Level | Notes |
|---|---|---|
| | Fast. No table rewrite. Metadata-only change. |
| | Fast. Non-volatile defaults stored in catalog, not backfilled. |
| | Fast. Column marked invisible; space reclaimed by VACUUM over time. |
| | Metadata change only. Does not touch existing rows. |
| | Non-blocking. Allows reads and writes during build. Slower than regular index creation. |
| | Non-blocking. Waits for queries using the index to finish, then drops. No table-level exclusive lock. |
| | Metadata change only. Fast. |
| | Metadata change only. Fast. |
| | Adds constraint for new rows only. Does not scan existing data. |
| | Scans existing rows but allows concurrent reads and writes. |
| | Brief catalog update. |
无论表大小如何,这些操作都能在毫秒级完成。它们仅在更新目录时短暂持有,不会重写数据。
AccessExclusiveLock| 操作 | 锁级别 | 说明 |
|---|---|---|
| | 速度快。 无需表重写,仅修改元数据。 |
| | 速度快。 非易失性默认值存储在目录中,无需回填。 |
| | 速度快。 列被标记为不可见,空间会随时间被VACUUM回收。 |
| | 仅修改元数据,不会触碰现有行。 |
| | 非阻塞。 构建期间允许读写。比普通索引创建慢。 |
| | 非阻塞。 等待使用该索引的查询完成后再删除,无表级排他锁。 |
| | 仅修改元数据,速度快。 |
| | 仅修改元数据,速度快。 |
| | 仅对新行生效的约束,不扫描现有数据。 |
| | 扫描现有行,但允许并发读写。 |
| | 短暂的目录更新。 |
Slow or Blocking Operations
慢速或阻塞操作
These rewrite the table or scan all rows. On large tables, they can lock out all access for seconds to hours.
| Operation | Lock Level | Why It's Slow |
|---|---|---|
| | Full table rewrite. Every row gets the computed value. |
| | Full table rewrite to convert stored data. |
| | Full table scan to verify no NULLs. See safe pattern below. |
| | Scans all rows to verify, blocks writes. |
| | Blocks writes for the entire build duration. |
| | Rewrites entire table in index order. |
| | Rewrites table to reclaim space. |
Key insight: blocks everything — reads and writes. Even if the operation itself is fast (milliseconds), it must wait for all in-flight transactions to finish before acquiring the lock. A long-running query or idle transaction can cause an to hang and queue up all subsequent queries behind it.
AccessExclusiveLockALTER TABLE这些操作会重写表或扫描所有行。在大表上,它们可能会阻塞所有访问数秒到数小时不等。
| 操作 | 锁级别 | 慢速原因 |
|---|---|---|
| | 全表重写,每一行都会被赋予计算值。 |
| | 全表重写以转换存储的数据。 |
| | 全表扫描以验证无NULL值。参见下方的安全模式。 |
| | 扫描所有行进行验证,阻塞写入。 |
| | 整个构建期间阻塞写入。 |
| | 按索引顺序重写整个表。 |
| | 重写表以回收空间。 |
核心要点: 会阻塞所有操作——包括读和写。即使操作本身很快(毫秒级),它也必须等待所有进行中的事务完成后才能获取锁。一个长时间运行的查询或空闲事务可能导致 挂起,并让所有后续查询在其后排队等待。
AccessExclusiveLockALTER TABLESafe Migration Patterns
安全迁移模式
Add a Column
添加列
sql
-- SAFE: nullable column, no default — instant
ALTER TABLE orders ADD COLUMN tracking_number TEXT;
-- SAFE (PG 11+): column with non-volatile default — instant
ALTER TABLE orders ADD COLUMN priority INTEGER NOT NULL DEFAULT 0;
-- UNSAFE: column with volatile default — full table rewrite
-- DON'T: ALTER TABLE orders ADD COLUMN created_at TIMESTAMPTZ DEFAULT now();
-- DO: add nullable, then backfill, then set default + NOT NULL
ALTER TABLE orders ADD COLUMN created_at TIMESTAMPTZ;
-- Backfill in batches (see Backfill section)
ALTER TABLE orders ALTER COLUMN created_at SET DEFAULT now();
ALTER TABLE orders ALTER COLUMN created_at SET NOT NULL; -- only if PG12+ or CHECK existssql
-- SAFE: nullable column, no default — instant
ALTER TABLE orders ADD COLUMN tracking_number TEXT;
-- SAFE (PG 11+): column with non-volatile default — instant
ALTER TABLE orders ADD COLUMN priority INTEGER NOT NULL DEFAULT 0;
-- UNSAFE: column with volatile default — full table rewrite
-- DON'T: ALTER TABLE orders ADD COLUMN created_at TIMESTAMPTZ DEFAULT now();
-- DO: add nullable, then backfill, then set default + NOT NULL
ALTER TABLE orders ADD COLUMN created_at TIMESTAMPTZ;
-- Backfill in batches (see Backfill section)
ALTER TABLE orders ALTER COLUMN created_at SET DEFAULT now();
ALTER TABLE orders ALTER COLUMN created_at SET NOT NULL; -- only if PG12+ or CHECK existsDrop a Column
删除列
sql
-- SAFE: instant (column marked invisible, space reclaimed by VACUUM)
ALTER TABLE orders DROP COLUMN old_status;Application coordination: Ensure your application no longer references the column before dropping it. For zero-downtime deploys, this requires two steps:
- Deploy code that doesn't read/write the column
- Then drop the column in a separate migration
Security caveat: does not physically delete the data. The column is marked as dropped in but the values remain on disk until reclaims the space — and even then, a superuser could recover them. If the column contains sensitive data, run on the table after dropping, or use dump/restore to ensure the data is truly gone.
DROP COLUMNpg_attributeVACUUMVACUUM FULLsql
-- SAFE: instant (column marked invisible, space reclaimed by VACUUM)
ALTER TABLE orders DROP COLUMN old_status;应用配合: 在删除列之前,请确保你的应用不再引用该列。对于零停机部署,这需要两个步骤:
- 部署不再读写该列的代码
- 然后在单独的迁移中删除该列
安全注意事项: 不会物理删除数据。列仅在中被标记为已删除,但其值仍保留在磁盘上,直到回收空间——即便如此,超级用户仍可以恢复数据。如果列包含敏感数据,请在删除后对表运行,或使用导出/导入的方式确保数据被彻底清除。
DROP COLUMNpg_attributeVACUUMVACUUM FULLRename a Column
重命名列
sql
-- SAFE: instant metadata change
ALTER TABLE orders RENAME COLUMN status TO order_status;Warning: This breaks any application code, views, or functions that reference the old column name. For zero-downtime deploys, use the column-swap pattern instead:
- Add the new column
- Deploy code that writes to both columns
- Backfill old rows
- Deploy code that reads from the new column
- Drop the old column
sql
-- SAFE: instant metadata change
ALTER TABLE orders RENAME COLUMN status TO order_status;警告: 这会破坏所有引用旧列名的应用代码、视图或函数。对于零停机部署,请改用列交换模式:
- 添加新列
- 部署同时写入两个列的代码
- 回填旧行数据
- 部署从新列读取的代码
- 删除旧列
Change a Column Type
修改列类型
Most type changes rewrite the entire table. Safe alternatives:
sql
-- UNSAFE: full table rewrite, blocks everything
-- DON'T: ALTER TABLE orders ALTER COLUMN amount TYPE NUMERIC(12,2);
-- SAFE: use a new column + backfill
ALTER TABLE orders ADD COLUMN amount_new NUMERIC(12,2);
-- Backfill in batches (see Backfill section below)
UPDATE orders SET amount_new = amount WHERE id BETWEEN 1 AND 10000;
-- ... continue in batches ...
-- Swap columns
ALTER TABLE orders DROP COLUMN amount;
ALTER TABLE orders RENAME COLUMN amount_new TO amount;Exception: Some casts don't require a rewrite and are fast:
| From | To | Rewrite? |
|---|---|---|
| No | Metadata only |
| No | Metadata only |
| No | Metadata only |
| Yes | Full rewrite |
| Yes | Full rewrite |
大多数类型变更会重写整个表。安全的替代方案:
sql
-- UNSAFE: full table rewrite, blocks everything
-- DON'T: ALTER TABLE orders ALTER COLUMN amount TYPE NUMERIC(12,2);
-- SAFE: use a new column + backfill
ALTER TABLE orders ADD COLUMN amount_new NUMERIC(12,2);
-- Backfill in batches (see Backfill section below)
UPDATE orders SET amount_new = amount WHERE id BETWEEN 1 AND 10000;
-- ... continue in batches ...
-- Swap columns
ALTER TABLE orders DROP COLUMN amount;
ALTER TABLE orders RENAME COLUMN amount_new TO amount;例外情况: 某些类型转换不需要重写,速度很快:
| 源类型 | 目标类型 | 是否需要重写? |
|---|---|---|
| 否 | 仅修改元数据 |
| 否 | 仅修改元数据 |
| 否 | 仅修改元数据 |
| 是 | 全表重写 |
| 是 | 全表重写 |
Add a NOT NULL Constraint
添加NOT NULL约束
sql
-- PG 18+: simplified two-step pattern
ALTER TABLE orders ALTER COLUMN order_status SET NOT NULL NOT VALID;
ALTER TABLE orders VALIDATE NOT NULL ON order_status;
-- PG 12–17: fast if a valid CHECK constraint already exists
-- Step 1: add CHECK (non-blocking scan)
ALTER TABLE orders ADD CONSTRAINT orders_status_nn CHECK (order_status IS NOT NULL) NOT VALID;
ALTER TABLE orders VALIDATE CONSTRAINT orders_status_nn;
-- Step 2: add NOT NULL (PG12+ recognizes the CHECK and skips the scan)
ALTER TABLE orders ALTER COLUMN order_status SET NOT NULL;
-- Step 3: drop the now-redundant CHECK
ALTER TABLE orders DROP CONSTRAINT orders_status_nn;
-- PG < 12: SET NOT NULL always scans the full table.
-- Ensure no NULLs exist first, then accept the brief lock.sql
-- PG 18+: simplified two-step pattern
ALTER TABLE orders ALTER COLUMN order_status SET NOT NULL NOT VALID;
ALTER TABLE orders VALIDATE NOT NULL ON order_status;
-- PG 12–17: fast if a valid CHECK constraint already exists
-- Step 1: add CHECK (non-blocking scan)
ALTER TABLE orders ADD CONSTRAINT orders_status_nn CHECK (order_status IS NOT NULL) NOT VALID;
ALTER TABLE orders VALIDATE CONSTRAINT orders_status_nn;
-- Step 2: add NOT NULL (PG12+ recognizes the CHECK and skips the scan)
ALTER TABLE orders ALTER COLUMN order_status SET NOT NULL;
-- Step 3: drop the now-redundant CHECK
ALTER TABLE orders DROP CONSTRAINT orders_status_nn;
-- PG < 12: SET NOT NULL always scans the full table.
-- Ensure no NULLs exist first, then accept the brief lock.Add a Foreign Key
添加外键
sql
-- UNSAFE: validates all existing rows while holding a heavy lock
-- DON'T: ALTER TABLE orders ADD CONSTRAINT fk_user FOREIGN KEY (user_id) REFERENCES users(id);
-- SAFE: two-step approach
-- Step 1: add without validation (blocks writes briefly, doesn't scan data)
ALTER TABLE orders ADD CONSTRAINT fk_user
FOREIGN KEY (user_id) REFERENCES users(id) NOT VALID;
-- Step 2: validate existing rows (allows concurrent reads and writes)
ALTER TABLE orders VALIDATE CONSTRAINT fk_user;sql
-- UNSAFE: validates all existing rows while holding a heavy lock
-- DON'T: ALTER TABLE orders ADD CONSTRAINT fk_user FOREIGN KEY (user_id) REFERENCES users(id);
-- SAFE: two-step approach
-- Step 1: add without validation (blocks writes briefly, doesn't scan data)
ALTER TABLE orders ADD CONSTRAINT fk_user
FOREIGN KEY (user_id) REFERENCES users(id) NOT VALID;
-- Step 2: validate existing rows (allows concurrent reads and writes)
ALTER TABLE orders VALIDATE CONSTRAINT fk_user;Add an Index
添加索引
sql
-- UNSAFE on large tables: blocks all writes for the entire build
-- DON'T: CREATE INDEX idx_orders_user ON orders (user_id);
-- SAFE: concurrent index creation (allows reads and writes)
CREATE INDEX CONCURRENTLY idx_orders_user ON orders (user_id);
-- IMPORTANT: if concurrent index creation fails (crashes, deadlock),
-- it leaves an INVALID index behind. Check and clean up:
SELECT indexrelname, idx_scan
FROM pg_stat_user_indexes
WHERE schemaname = 'public'
AND indexrelname = 'idx_orders_user';
-- Check for invalid indexes
SELECT indexrelid::regclass AS index_name, indisvalid
FROM pg_index
WHERE NOT indisvalid;
-- Drop and retry if invalid
DROP INDEX CONCURRENTLY idx_orders_user;
CREATE INDEX CONCURRENTLY idx_orders_user ON orders (user_id);sql
-- UNSAFE on large tables: blocks all writes for the entire build
-- DON'T: CREATE INDEX idx_orders_user ON orders (user_id);
-- SAFE: concurrent index creation (allows reads and writes)
CREATE INDEX CONCURRENTLY idx_orders_user ON orders (user_id);
-- IMPORTANT: if concurrent index creation fails (crashes, deadlock),
-- it leaves an INVALID index behind. Check and clean up:
SELECT indexrelname, idx_scan
FROM pg_stat_user_indexes
WHERE schemaname = 'public'
AND indexrelname = 'idx_orders_user';
-- Check for invalid indexes
SELECT indexrelid::regclass AS index_name, indisvalid
FROM pg_index
WHERE NOT indisvalid;
-- Drop and retry if invalid
DROP INDEX CONCURRENTLY idx_orders_user;
CREATE INDEX CONCURRENTLY idx_orders_user ON orders (user_id);Add a Unique Constraint
添加唯一约束
sql
-- A UNIQUE constraint creates an index. Use CONCURRENTLY to avoid blocking:
-- Step 1: create a unique index concurrently
CREATE UNIQUE INDEX CONCURRENTLY idx_orders_tracking_uniq ON orders (tracking_number);
-- Step 2: attach it as a constraint (instant)
ALTER TABLE orders ADD CONSTRAINT orders_tracking_uniq UNIQUE USING INDEX idx_orders_tracking_uniq;sql
-- A UNIQUE constraint creates an index. Use CONCURRENTLY to avoid blocking:
-- Step 1: create a unique index concurrently
CREATE UNIQUE INDEX CONCURRENTLY idx_orders_tracking_uniq ON orders (tracking_number);
-- Step 2: attach it as a constraint (instant)
ALTER TABLE orders ADD CONSTRAINT orders_tracking_uniq UNIQUE USING INDEX idx_orders_tracking_uniq;Redefine a Primary Key
重新定义主键
Redefining a PK (e.g., switching from to a composite key, or from to ) requires both a UNIQUE constraint and NOT NULL — both of which can cause long-lasting locks if done naively. The zero-downtime approach builds each ingredient separately:
idintbigintsql
-- Step 1: add CHECK NOT NULL constraint without validation (brief lock)
ALTER TABLE orders ADD CONSTRAINT orders_new_id_nn
CHECK (new_id IS NOT NULL) NOT VALID;
-- Step 2: validate existing rows (allows concurrent reads and writes)
ALTER TABLE orders VALIDATE CONSTRAINT orders_new_id_nn;
-- Step 3: build unique index concurrently (non-blocking)
CREATE UNIQUE INDEX CONCURRENTLY idx_orders_new_pkey
ON orders (new_id);
-- Step 4: drop the old PK
ALTER TABLE orders DROP CONSTRAINT orders_pkey;
-- Step 5: add new PK using the existing index (instant — also implicitly adds NOT NULL)
ALTER TABLE orders ADD CONSTRAINT orders_pkey
PRIMARY KEY USING INDEX idx_orders_new_pkey;
-- Step 6: drop the now-redundant CHECK constraint
ALTER TABLE orders DROP CONSTRAINT orders_new_id_nn;Why this works: Step 5 is fast because Postgres reuses the already-built unique index and recognizes the existing CHECK constraint, skipping both the index build and the full-table NOT NULL scan (PG12+).
重新定义主键(例如从切换到复合键,或从切换到)需要同时有唯一约束和NOT NULL——如果操作不当,两者都可能导致长时间锁表。零停机方案是分别构建每个组件:
idintbigintsql
-- Step 1: add CHECK NOT NULL constraint without validation (brief lock)
ALTER TABLE orders ADD CONSTRAINT orders_new_id_nn
CHECK (new_id IS NOT NULL) NOT VALID;
-- Step 2: validate existing rows (allows concurrent reads and writes)
ALTER TABLE orders VALIDATE CONSTRAINT orders_new_id_nn;
-- Step 3: build unique index concurrently (non-blocking)
CREATE UNIQUE INDEX CONCURRENTLY idx_orders_new_pkey
ON orders (new_id);
-- Step 4: drop the old PK
ALTER TABLE orders DROP CONSTRAINT orders_pkey;
-- Step 5: add new PK using the existing index (instant — also implicitly adds NOT NULL)
ALTER TABLE orders ADD CONSTRAINT orders_pkey
PRIMARY KEY USING INDEX idx_orders_new_pkey;
-- Step 6: drop the now-redundant CHECK constraint
ALTER TABLE orders DROP CONSTRAINT orders_new_id_nn;原理: 第5步速度很快,因为Postgres会复用已构建好的唯一索引,并且识别出现有的CHECK约束,从而跳过索引构建和全表NOT NULL扫描(PG12+)。
Drop a Constraint
删除约束
sql
-- SAFE: instant metadata change
ALTER TABLE orders DROP CONSTRAINT orders_tracking_uniq;
-- If dropping a FK that has a supporting index you no longer need:
ALTER TABLE orders DROP CONSTRAINT fk_user;
DROP INDEX idx_orders_user_id; -- only if no other queries use itsql
-- SAFE: instant metadata change
ALTER TABLE orders DROP CONSTRAINT orders_tracking_uniq;
-- If dropping a FK that has a supporting index you no longer need:
ALTER TABLE orders DROP CONSTRAINT fk_user;
DROP INDEX idx_orders_user_id; -- only if no other queries use itBackfill Strategies
回填策略
Always backfill in batches — never in a single UPDATE. See backfill-strategies for batch-by-PK patterns, resumable progress tracking, and tuning guidance.
始终分批回填——绝不要用单条UPDATE完成。请参阅 backfill-strategies 了解按主键分批的模式、可恢复的进度跟踪以及调优指南。
Migration Validation
迁移验证
Run validation queries before and after every migration. See validation-queries for the full set of checks: NULL detection, duplicate detection, orphan rows, cast failures, duration estimation, schema verification, data integrity, and query performance.
在每次迁移前后运行验证查询。请参阅 validation-queries 获取完整的检查项:NULL检测、重复检测、孤立行、类型转换失败、耗时估算、schema验证、数据完整性以及查询性能。
Rollback Planning
回滚规划
Every migration should have a rollback plan documented before execution.
每次迁移在执行前都应记录回滚计划。
Reversible Operations
可回滚操作
| Operation | Rollback |
|---|---|
| |
| |
| |
| |
| |
| Cannot directly undo — need to re-add old column and backfill from a backup |
| 操作 | 回滚方式 |
|---|---|
| |
| |
| |
| |
| |
| 无法直接撤销——需要重新添加旧列并从备份回填 |
Irreversible Operations
不可逆操作
These require restoring from a backup or the database fork to undo:
- — data is gone once VACUUM reclaims it
DROP COLUMN - with lossy cast (e.g.,
ALTER COLUMN TYPE→NUMERIC,INTEGER→TEXT)VARCHAR(50) - /
DELETEduring data cleanupTRUNCATE DROP TABLE
This is where a database fork is invaluable. If you forked before the migration, the original database has the pre-migration state. If the migration went wrong, your production data is untouched — just delete the fork and start over.
这些操作需要从备份或数据库fork恢复才能撤销:
- —— 一旦VACUUM回收空间,数据就会丢失
DROP COLUMN - 带有损转换的(例如
ALTER COLUMN TYPE→NUMERIC、INTEGER→TEXT)VARCHAR(50) - 数据清理期间的/
DELETETRUNCATE DROP TABLE
这就是数据库fork的价值所在。 如果你在迁移前fork了数据库,原始数据库仍保留迁移前的状态。如果迁移出错,你的生产数据不会受到任何影响——只需删除fork重新开始即可。
Transaction Strategy
事务策略
There are two approaches for executing multiple DDL statements. Each has tradeoffs:
Wrapped in one transaction — all changes succeed or all roll back. Use this when atomicity matters more than lock duration, and all operations are fast (milliseconds).
sql
BEGIN;
ALTER TABLE orders ADD COLUMN priority INTEGER NOT NULL DEFAULT 0;
ALTER TABLE orders ADD COLUMN tags TEXT[] NOT NULL DEFAULT '{}';
CREATE INDEX ON orders USING GIN (tags);
ALTER TABLE orders DROP COLUMN old_priority;
-- Verify before committing
SELECT column_name, data_type
FROM information_schema.columns
WHERE table_name = 'orders'
ORDER BY ordinal_position;
COMMIT;
-- Or ROLLBACK; if something looks wrongSeparate transactions — each DDL runs and commits independently. Use this when lock duration matters more than atomicity. In a single transaction, all locks are held until — so if you have 5 DDL statements, the from the first one blocks traffic for the entire duration of all 5. Separate transactions release locks between statements.
COMMITAccessExclusiveLocksql
-- Each statement auto-commits
ALTER TABLE orders ADD COLUMN priority INTEGER NOT NULL DEFAULT 0;
ALTER TABLE orders ADD COLUMN tags TEXT[] NOT NULL DEFAULT '{}';
ALTER TABLE orders DROP COLUMN old_priority;The tradeoff: separate transactions can leave the schema in a partially migrated state if a later statement fails. You'll need a rollback plan for each step individually.
Cannot use transactions with:
- (explicitly disallowed inside a transaction)
CREATE INDEX CONCURRENTLY DROP INDEX CONCURRENTLY- Any statement that requires its own transaction context
执行多条DDL语句有两种方法,各有优劣:
包裹在单个事务中 —— 所有变更要么全部成功,要么全部回滚。当原子性比锁持续时间更重要,且所有操作都很快(毫秒级)时使用此方式。
sql
BEGIN;
ALTER TABLE orders ADD COLUMN priority INTEGER NOT NULL DEFAULT 0;
ALTER TABLE orders ADD COLUMN tags TEXT[] NOT NULL DEFAULT '{}';
CREATE INDEX ON orders USING GIN (tags);
ALTER TABLE orders DROP COLUMN old_priority;
-- Verify before committing
SELECT column_name, data_type
FROM information_schema.columns
WHERE table_name = 'orders'
ORDER BY ordinal_position;
COMMIT;
-- Or ROLLBACK; if something looks wrong独立事务 —— 每个DDL独立运行并提交。当锁持续时间比原子性更重要时使用此方式。在单个事务中,所有锁都会一直持有到——所以如果你有5条DDL语句,第一条语句的会在所有5条语句的整个执行期间阻塞流量。独立事务会在语句之间释放锁。
COMMITAccessExclusiveLocksql
-- Each statement auto-commits
ALTER TABLE orders ADD COLUMN priority INTEGER NOT NULL DEFAULT 0;
ALTER TABLE orders ADD COLUMN tags TEXT[] NOT NULL DEFAULT '{}';
ALTER TABLE orders DROP COLUMN old_priority;权衡之处: 如果后续语句失败,独立事务可能会让schema处于部分迁移的状态。你需要为每个步骤单独制定回滚计划。
以下操作不能在事务中使用:
- (明确禁止在事务内运行)
CREATE INDEX CONCURRENTLY DROP INDEX CONCURRENTLY- 任何需要自身事务上下文的语句
Dealing with Long-Running Queries
处理长时间运行的查询
A fast can still hang if it's waiting to acquire behind a long-running query. Worse, the waiting DDL blocks all subsequent queries too — even simple SELECTs pile up behind it:
ALTER TABLEAccessExclusiveLockSession 1: SELECT COUNT(*) FROM orders; -- long query, holds AccessShareLock
Session 2: ALTER TABLE orders ADD COLUMN ...; -- waits for Session 1 (needs AccessExclusiveLock)
Session 3: SELECT * FROM orders WHERE id = 123; -- BLOCKED by Session 2's lock queue entry
Session 4: INSERT INTO orders (...) VALUES (...); -- also BLOCKED
-- All sessions freeze until Session 1 finishes and Session 2 completes or times outThis is why is critical — without it, a single slow query can cascade into an application-wide outage.
lock_timeout即使是快速的,如果它在一个长时间运行的查询后等待获取,也可能挂起。更糟糕的是,等待中的DDL会阻塞所有后续查询——哪怕是简单的SELECT也会在其后堆积:
ALTER TABLEAccessExclusiveLockSession 1: SELECT COUNT(*) FROM orders; -- long query, holds AccessShareLock
Session 2: ALTER TABLE orders ADD COLUMN ...; -- waits for Session 1 (needs AccessExclusiveLock)
Session 3: SELECT * FROM orders WHERE id = 123; -- BLOCKED by Session 2's lock queue entry
Session 4: INSERT INTO orders (...) VALUES (...); -- also BLOCKED
-- All sessions freeze until Session 1 finishes and Session 2 completes or times out这就是为什么 至关重要——没有它,一个慢查询可能会级联导致整个应用中断。
lock_timeoutSet Timeouts
设置超时
lock_timeoutALTER TABLEstatement_timeoutCREATE INDEX CONCURRENTLYstatement_timeoutChoosing timeout values:
There are two schools of thought:
- Conservative (50-100ms lock_timeout, hundreds of retries): Minimizes the window where a waiting DDL blocks other queries. Each attempt is nearly invisible to application traffic, but requires retry logic. Best for high-traffic OLTP systems where even a few seconds of blocked queries is unacceptable.
- Pragmatic (3-5s lock_timeout, few retries): Gives the lock a reasonable chance to be acquired on each attempt, reducing the need for complex retry logic. Acceptable for most applications where brief pauses are tolerable.
Pick based on your traffic profile: the higher your query throughput, the shorter your should be — because even a brief queue-up affects more queries per second. For , set it to a generous multiple of what you expect the operation to take (e.g., 30s for metadata-only changes, minutes for VALIDATE CONSTRAINT on large tables, disabled for CREATE INDEX CONCURRENTLY).
lock_timeoutstatement_timeoutsql
-- Fail fast instead of blocking all queries behind you
SET lock_timeout = '5s';
SET statement_timeout = '30s';
ALTER TABLE orders ADD COLUMN tracking_number TEXT;
-- If it fails with "canceling statement due to lock timeout":
-- 1. Find what's blocking
SELECT pid, state, query, now() - query_start AS duration
FROM pg_stat_activity
WHERE state != 'idle'
ORDER BY duration DESC;
-- 2. Wait for the blocker to finish, or cancel it if appropriate
-- SELECT pg_cancel_backend(<pid>);
-- 3. Retry the ALTER TABLE
SET lock_timeout = '5s';
ALTER TABLE orders ADD COLUMN tracking_number TEXT;
-- Reset timeouts when done
RESET lock_timeout;
RESET statement_timeout;lock_timeoutALTER TABLEstatement_timeoutCREATE INDEX CONCURRENTLYstatement_timeout如何选择超时值:
有两种主流思路:
- 保守策略(50-100ms lock_timeout,数百次重试): 最小化等待中的DDL阻塞其他查询的时间窗口。每次尝试对应用流量几乎不可见,但需要重试逻辑。最适合高流量OLTP系统,这类系统中即使几秒的查询阻塞也是不可接受的。
- 实用策略(3-5s lock_timeout,少量重试): 给每次尝试合理的锁获取机会,减少对复杂重试逻辑的需求。对于可以容忍短暂停顿的大多数应用来说是可接受的。
根据你的流量情况选择:查询吞吐量越高, 应该越短——因为即使是短暂的排队,每秒影响的查询数也会更多。对于,将其设置为你预期操作耗时的较大倍数(例如,元数据-only操作设为30秒,大表上的VALIDATE CONSTRAINT设为数分钟,CREATE INDEX CONCURRENTLY则禁用)。
lock_timeoutstatement_timeoutsql
-- Fail fast instead of blocking all queries behind you
SET lock_timeout = '5s';
SET statement_timeout = '30s';
ALTER TABLE orders ADD COLUMN tracking_number TEXT;
-- If it fails with "canceling statement due to lock timeout":
-- 1. Find what's blocking
SELECT pid, state, query, now() - query_start AS duration
FROM pg_stat_activity
WHERE state != 'idle'
ORDER BY duration DESC;
-- 2. Wait for the blocker to finish, or cancel it if appropriate
-- SELECT pg_cancel_backend(<pid>);
-- 3. Retry the ALTER TABLE
SET lock_timeout = '5s';
ALTER TABLE orders ADD COLUMN tracking_number TEXT;
-- Reset timeouts when done
RESET lock_timeout;
RESET statement_timeout;The Retry-With-Timeout Pattern
带超时的重试模式
For automated migration runners, wrap DDL in a retry loop with a short lock timeout:
sql
DO $$
DECLARE
max_attempts INTEGER := 5;
attempt INTEGER := 1;
success BOOLEAN := FALSE;
BEGIN
WHILE attempt <= max_attempts AND NOT success LOOP
BEGIN
SET lock_timeout = '3s';
-- Replace with your DDL statement
ALTER TABLE orders ADD COLUMN tracking_number TEXT;
success := TRUE;
RAISE NOTICE 'DDL succeeded on attempt %', attempt;
EXCEPTION
WHEN lock_not_available THEN
RAISE NOTICE 'Attempt % failed (lock not available), retrying...', attempt;
PERFORM pg_sleep(2 * attempt); -- linear backoff
attempt := attempt + 1;
END;
END LOOP;
IF NOT success THEN
RAISE EXCEPTION 'DDL failed after % attempts', max_attempts;
END IF;
END $$;This prevents the migration from creating a pile-up of blocked queries behind it. Each attempt either succeeds quickly or gives up and lets normal traffic flow.
Alternative: — For the highest-traffic systems, use to test lock availability before running DDL. Unlike , fails instantly without ever entering the lock queue, so there is zero risk of cascading blocked queries. The tradeoff is more retries:
NOWAITLOCK TABLE ... NOWAITlock_timeoutNOWAITsql
BEGIN;
LOCK TABLE orders IN ACCESS EXCLUSIVE MODE NOWAIT;
-- If we get here, we have the lock — run DDL
ALTER TABLE orders ADD COLUMN tracking_number TEXT;
COMMIT;
-- If LOCK fails with "could not obtain lock", retry after a short sleep对于自动化迁移执行器,可以将DDL包裹在带有短锁超时的重试循环中:
sql
DO $$
DECLARE
max_attempts INTEGER := 5;
attempt INTEGER := 1;
success BOOLEAN := FALSE;
BEGIN
WHILE attempt <= max_attempts AND NOT success LOOP
BEGIN
SET lock_timeout = '3s';
-- Replace with your DDL statement
ALTER TABLE orders ADD COLUMN tracking_number TEXT;
success := TRUE;
RAISE NOTICE 'DDL succeeded on attempt %', attempt;
EXCEPTION
WHEN lock_not_available THEN
RAISE NOTICE 'Attempt % failed (lock not available), retrying...', attempt;
PERFORM pg_sleep(2 * attempt); -- linear backoff
attempt := attempt + 1;
END;
END LOOP;
IF NOT success THEN
RAISE EXCEPTION 'DDL failed after % attempts', max_attempts;
END IF;
END $$;这可以防止迁移在其后方造成大量阻塞查询。每次尝试要么快速成功,要么放弃并让正常流量继续通行。
替代方案: —— 对于最高流量的系统,在运行DDL前使用 测试锁的可用性。与 不同, 会立即失败,不会进入锁队列,因此完全没有级联阻塞查询的风险。权衡之处是需要更多次重试:
NOWAITLOCK TABLE ... NOWAITlock_timeoutNOWAITsql
BEGIN;
LOCK TABLE orders IN ACCESS EXCLUSIVE MODE NOWAIT;
-- If we get here, we have the lock — run DDL
ALTER TABLE orders ADD COLUMN tracking_number TEXT;
COMMIT;
-- If LOCK fails with "could not obtain lock", retry after a short sleepFork-Based Migration Testing
基于Fork的迁移测试
The safest way to test a migration is to run it against a copy of your actual database — same schema, same data, same edge cases. Providers such as Neon support fast database forking. Without database forking, you need to manually dump and restore your database, which can take a long time for large datasets.
测试迁移最安全的方式是在真实数据库的副本上运行——相同的schema、相同的数据、相同的边界情况。Neon 等服务商支持快速数据库fork。如果没有数据库fork功能,你需要手动导出并恢复数据库,对于大数据集这可能会花费很长时间。
With Forking
使用Fork的测试方法
- Fork your database — create a full copy using your provider's fork feature (takes seconds)
- Inspect the current schema on the fork to confirm it matches production
- Run your migration on the fork
- Validate — run your checks (see Pre/Post-Migration Validation sections above)
- If it worked: apply the same migration to production
- If it failed: delete the fork — your production database is untouched
This catches problems that never show up in empty test databases:
- Data that violates a new constraint
- Type casts that fail on real values
- Migrations that are fast on 100 rows but lock the table for minutes on 10 million
- Index creation that runs out of memory or disk space
Limitation: fork-based testing runs your migration in isolation — it won't catch issues caused by concurrent database traffic (e.g., lock contention under load, deadlocks with concurrent writes, or replication lag from heavy WAL generation). For most applications, fork-based testing is sufficient. For very high-uptime applications, use PgDog's mirroring feature to replay production traffic against the fork — it reproduces queries byte-for-byte with realistic timing, and you can filter to DDL-only or DML-only and control exposure percentage to ramp up gradually.
- Fork你的数据库 —— 使用服务商的fork功能创建完整副本(仅需几秒)
- 检查fork上的当前schema,确认与生产环境一致
- 在fork上运行你的迁移
- 验证 —— 运行你的检查项(参见上文的迁移前后验证部分)
- 如果成功: 将相同的迁移应用到生产环境
- 如果失败: 删除fork——你的生产数据库不会受到任何影响
这可以发现那些在空测试数据库上永远不会出现的问题:
- 违反新约束的数据
- 在真实值上失败的类型转换
- 在100行上运行很快但在1000万行上锁表数分钟的迁移
- 耗尽内存或磁盘空间的索引创建
局限性: 基于fork的测试是在隔离环境中运行迁移——它无法发现由并发数据库流量导致的问题(例如,负载下的锁争用、并发写入导致的死锁,或大量WAL生成导致的复制延迟)。对于大多数应用,基于fork的测试已经足够。对于要求极高可用性的应用,可以使用 PgDog 的镜像功能将生产流量重放到fork上——它会逐字节复现查询并保留真实的时间特性,你还可以过滤为仅DDL或仅DML,并控制流量百分比逐步放量。
Without Forking
无Fork时的测试方案
Create a test database from a backup or dump:
bash
undefined从备份或转储创建测试数据库:
bash
undefinedDump your production database
Dump your production database
pg_dump -Fc my_app_db > backup.dump
pg_dump -Fc my_app_db > backup.dump
Restore into a test database
Restore into a test database
createdb migration_test
pg_restore -d migration_test backup.dump
createdb migration_test
pg_restore -d migration_test backup.dump
Or clone from a live database (requires downtime on source during copy)
Or clone from a live database (requires downtime on source during copy)
createdb migration_test -T my_app_db
undefinedcreatedb migration_test -T my_app_db
undefinedComplete Migration Example
完整迁移示例
For a full end-to-end walkthrough (plan, fork, run, validate, apply, clean up), see complete-example.
有关完整的端到端演练(规划、fork、运行、验证、应用、清理),请参阅 complete-example。
Advanced Considerations
高级注意事项
Subtransactions in PL/pgSQL retry loops: The block in the retry-with-timeout pattern creates implicit subtransactions. Under high write throughput, this can trigger SubtransSLRU contention on replicas — especially if the retry loop runs as a long-lived transaction with many attempts. If you see replica lag during retries, move the retry logic to the application layer (separate transactions per attempt) instead of using PL/pgSQL exception handling.
BEGIN/EXCEPTION WHEN/ENDAutovacuum can block VALIDATE CONSTRAINT: acquires , which conflicts with autovacuum running in transaction ID wraparound prevention mode. If hangs unexpectedly, check for autovacuum processes on the same table. You may need to wait for wraparound-prevention autovacuum to finish — do not cancel it, as that can lead to data loss if the table approaches the XID wraparound limit.
VALIDATE CONSTRAINTShareUpdateExclusiveLockVALIDATEpg_stat_activityPL/pgSQL重试循环中的子事务: 带超时重试模式中的 块会创建隐式子事务。在高写入吞吐量下,这可能会在副本上触发SubtransSLRU争用——尤其是当重试循环作为一个长生命周期事务运行且有多次尝试时。如果你在重试期间看到副本延迟,请将重试逻辑移到应用层(每次尝试使用独立事务),而不是使用PL/pgSQL异常处理。
BEGIN/EXCEPTION WHEN/ENDAutovacuum可能阻塞VALIDATE CONSTRAINT: 会获取 ,这与运行在事务ID回卷预防模式下的autovacuum冲突。如果 意外挂起,请检查 中同一张表上的autovacuum进程。你可能需要等待回卷预防的autovacuum完成——不要取消它,因为如果表接近XID回卷限制,可能会导致数据丢失。
VALIDATE CONSTRAINTShareUpdateExclusiveLockVALIDATEpg_stat_activityCommon Pitfalls
常见陷阱
- Testing migrations on empty tables — a migration that runs in 1ms on an empty table can lock a 10M-row table for minutes. Always test against realistic data volumes.
- Forgetting on index creation —
CONCURRENTLY(withoutCREATE INDEX) blocks all writes. On a table with active traffic, this causes downtime.CONCURRENTLY - Adding NOT NULL without the two-step pattern — on large tables in PG < 12, scans every row while holding
SET NOT NULL. Use the CHECK constraint pattern.AccessExclusiveLock - No lock timeout — a fast ALTER TABLE can block behind a long-running query, and every subsequent query stacks up behind it. Always for production DDL.
SET lock_timeout - Backfilling in one big transaction — a single on 10M rows generates enormous WAL, bloats the table, and holds locks for the entire duration. Always batch.
UPDATE orders SET x = y - Leaving invalid indexes behind — if fails, it leaves an invisible invalid index that consumes space and slows writes. Check
CREATE INDEX CONCURRENTLYafter every concurrent index operation.pg_index.indisvalid - Dropping columns before updating application code — in a running system, the old code still references the column. Deploy the code change first, then drop the column in a subsequent migration.
- Not checking replication lag — large backfills generate heavy WAL. If you have read replicas, monitor during and after the migration.
pg_stat_replication - Assuming ALTER COLUMN TYPE is safe — most type changes rewrite the entire table. Use the add-new-column + backfill + swap pattern for large tables.
- 在空表上测试迁移 —— 在空表上1ms运行完的迁移,在1000万行的表上可能锁表数分钟。始终使用真实数据量进行测试。
- 创建索引时忘记加—— (不带
CONCURRENTLY的)CONCURRENTLY会阻塞所有写入。在有活跃流量的表上,这会导致停机。CREATE INDEX - 不使用两步模式添加NOT NULL —— 在PG < 12的大表上,会在持有
SET NOT NULL的同时扫描每一行。请使用CHECK约束模式。AccessExclusiveLock - 没有设置锁超时 —— 快速的ALTER TABLE可能在长时间运行的查询后阻塞,所有后续查询都会在其后堆积。生产环境DDL务必设置。
lock_timeout - 在一个大事务中回填 —— 在1000万行上运行单条会生成巨大的WAL,导致表膨胀,并且在整个期间持有锁。始终分批操作。
UPDATE orders SET x = y - 遗留无效索引 —— 如果失败,会留下一个不可见的无效索引,占用空间并拖慢写入。每次并发索引操作后都要检查
CREATE INDEX CONCURRENTLY。pg_index.indisvalid - 在更新应用代码前删除列 —— 在运行中的系统里,旧代码仍在引用该列。请先部署代码变更,再在后续迁移中删除列。
- 不检查复制延迟 —— 大量回填会生成很重的WAL。如果你有读副本,请在迁移期间和之后监控。
pg_stat_replication - 假设ALTER COLUMN TYPE是安全的 —— 大多数类型变更会重写整个表。对于大表,请使用“添加新列+回填+交换”的模式。