go-data-persistence

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

Data Persistence in Go

Go语言中的数据持久化

Quick Reference

快速参考

TopicReference
Connection pool internals, sizing, pgx pools, monitoringreferences/connection-pooling.md
golang-migrate setup, file conventions, CI/CD integrationreferences/migrations.md
Transaction helpers, service-layer transactions, isolation levelsreferences/transactions.md
主题参考文档
连接池内部机制、大小配置、pgx连接池、监控references/connection-pooling.md
golang-migrate配置、文件规范、CI/CD集成references/migrations.md
事务工具、服务层事务、隔离级别references/transactions.md

Choosing Your Approach

选择合适的实现方式

Pick the right tool based on your project's needs:
FactorRaw SQL (sqlx/pgx)ORM (Ent/GORM)
Complex queriesPreferredAwkward
Type safetyManualAuto-generated
Performance controlFullLimited
Rapid prototypingSlowerFaster
Schema migrationsgolang-migrateBuilt-in (Ent)
Learning curveSQL knowledgeORM API
根据项目需求选择合适的工具:
考量因素原生SQL(sqlx/pgx)ORM框架(Ent/GORM)
复杂查询推荐使用使用不便
类型安全手动维护自动生成
性能控制完全可控受限
快速原型开发速度较慢速度较快
模式迁移依赖golang-migrate内置支持(Ent)
学习曲线需要SQL知识需要掌握ORM API

When to Use Raw SQL (sqlx/pgx)

何时使用原生SQL(sqlx/pgx)

  • You need full control over query performance and execution plans
  • Your domain has complex joins, CTEs, window functions, or recursive queries
  • You want zero abstraction overhead and direct access to PostgreSQL features
  • Your team is comfortable writing and maintaining SQL
  • You need advanced PostgreSQL features like
    LISTEN/NOTIFY
    , advisory locks, or
    COPY
pgx is the recommended PostgreSQL driver for Go. It provides native PostgreSQL protocol support, better performance than
database/sql
, and access to PostgreSQL-specific features. Use sqlx when you need
database/sql
compatibility or work with multiple database backends.
  • 需要完全控制查询性能和执行计划
  • 业务领域涉及复杂关联查询、CTE(公共表表达式)、窗口函数或递归查询
  • 希望零抽象开销,直接访问PostgreSQL特性
  • 团队熟悉SQL编写与维护
  • 需要PostgreSQL高级特性,如
    LISTEN/NOTIFY
    、 advisory locks或
    COPY
pgx是Go语言推荐的PostgreSQL驱动,它提供原生PostgreSQL协议支持,性能优于
database/sql
,并能访问PostgreSQL专属特性。当需要兼容
database/sql
或同时操作多种数据库后端时,使用sqlx

When to Use an ORM (Ent/GORM)

何时使用ORM框架(Ent/GORM)

  • You want type-safe, generated query builders and avoid writing SQL
  • Your schema is mostly CRUD with straightforward relationships
  • You value generated code, schema-as-code, and automatic migrations (Ent)
  • You are prototyping quickly and want to iterate on the schema fast
Ent is preferred over GORM for new projects. It uses code generation for type safety, has a declarative schema DSL, built-in migration support, and integrates with GraphQL. GORM is suitable if the team already knows it or if the project is small.
  • 希望使用类型安全的自动生成查询构建器,避免编写SQL
  • 数据模型以CRUD操作为主,关系简单
  • 重视代码生成、代码化模式和自动迁移(Ent)
  • 需要快速原型开发,快速迭代数据模型
对于新项目,优先选择Ent而非GORM。它通过代码生成保证类型安全,拥有声明式模式DSL、内置迁移支持,并能与GraphQL集成。如果团队已熟悉GORM或项目规模较小,GORM也是合适的选择。

Connection Setup

连接配置

Every Go application connecting to a database needs a properly configured connection pool. The
database/sql
package manages pooling automatically, but the defaults are not suitable for production.
go
db, err := sql.Open("postgres", connStr)
if err != nil {
    return fmt.Errorf("opening db: %w", err)
}

// Connection pool configuration
db.SetMaxOpenConns(25)                 // Max simultaneous connections
db.SetMaxIdleConns(10)                 // Connections kept alive when idle
db.SetConnMaxLifetime(5 * time.Minute) // Recycle connections
db.SetConnMaxIdleTime(1 * time.Minute) // Close idle connections

// Verify connection
if err := db.PingContext(ctx); err != nil {
    return fmt.Errorf("pinging db: %w", err)
}
所有连接数据库的Go应用都需要正确配置连接池。
database/sql
包会自动管理连接池,但默认配置不适合生产环境。
go
db, err := sql.Open("postgres", connStr)
if err != nil {
    return fmt.Errorf("opening db: %w", err)
}

// 连接池配置
db.SetMaxOpenConns(25)                 // 最大同时连接数
db.SetMaxIdleConns(10)                 // 空闲时保持的连接数
db.SetConnMaxLifetime(5 * time.Minute) // 连接最大复用时长
db.SetConnMaxIdleTime(1 * time.Minute) // 连接最大空闲时长

// 验证连接
if err := db.PingContext(ctx); err != nil {
    return fmt.Errorf("pinging db: %w", err)
}

Pool Settings Explained

连接池参数说明

MaxOpenConns -- The maximum number of open connections to the database. This prevents your application from overwhelming the database with too many concurrent connections. Set to approximately 25 for typical web apps. To calculate: divide your database's
max_connections
(minus a reserve for admin and replication) by the number of application instances. If your DB allows 100 connections, you have 3 app instances, and you reserve 10 for admin, set this to
(100 - 10) / 3 = 30
.
MaxIdleConns -- The number of connections kept alive in the pool when not in use. These warm connections avoid the latency of establishing new connections for each request. Set to approximately 10, or roughly 40% of
MaxOpenConns
. Setting this too high wastes database connections; setting it too low causes frequent reconnections.
ConnMaxLifetime -- The maximum amount of time a connection can be reused. After this duration, the connection is closed and a new one is created on the next request. This helps pick up DNS changes (important for cloud databases that failover to new IPs), rebalance load across read replicas, and prevent connections from becoming stale. A value of 5 minutes is typical. Set shorter (1-2 min) if your infrastructure uses DNS-based failover.
ConnMaxIdleTime -- The maximum amount of time a connection can sit idle before it is closed. This releases connections back to the database during low-traffic periods, freeing resources. A value of 1 minute is typical. This should be shorter than
ConnMaxLifetime
.
For pgx-specific pooling with native PostgreSQL support, see references/connection-pooling.md.
MaxOpenConns -- 数据库的最大打开连接数。该参数可防止应用创建过多并发连接导致数据库过载。典型Web应用设置为25左右。计算方式:将数据库的
max_connections
(减去管理员和复制预留的连接数)除以应用实例数。例如,数据库允许100个连接,有3个应用实例,预留10个管理员连接,则设置为
(100 - 10) / 3 = 30
MaxIdleConns -- 连接池中空闲时保持的连接数。这些预热后的连接可避免每次请求都建立新连接的延迟。通常设置为10左右,或约为
MaxOpenConns
的40%。设置过高会浪费数据库连接资源;设置过低会导致频繁重建连接。
ConnMaxLifetime -- 连接的最大复用时长。超过该时长后,连接会被关闭,下一次请求时会创建新连接。这有助于获取DNS变更(对故障转移到新IP的云数据库尤为重要)、在只读副本间重新平衡负载,以及防止连接失效。典型值为5分钟。如果基础设施使用基于DNS的故障转移,可设置更短(1-2分钟)。
ConnMaxIdleTime -- 连接的最大空闲时长。超过该时长后,空闲连接会被关闭。这能在低流量时段将连接释放回数据库,节省资源。典型值为1分钟。该值应短于
ConnMaxLifetime
如需了解支持原生PostgreSQL特性的pgx专属连接池配置,请查看references/connection-pooling.md

Repository Pattern

仓储模式

Define a store interface at the consumer for testability. Implement against a concrete database driver. This pattern keeps your domain logic decoupled from the database.
go
// Store interface for testability
type UserStore interface {
    GetUser(ctx context.Context, id string) (*User, error)
    ListUsers(ctx context.Context, limit, offset int) ([]*User, error)
    CreateUser(ctx context.Context, u *User) error
}

// sqlx implementation
type PostgresUserStore struct {
    db *sqlx.DB
}

func NewPostgresUserStore(db *sqlx.DB) *PostgresUserStore {
    return &PostgresUserStore{db: db}
}

func (s *PostgresUserStore) GetUser(ctx context.Context, id string) (*User, error) {
    var u User
    err := s.db.GetContext(ctx, &u, "SELECT * FROM users WHERE id = $1", id)
    if errors.Is(err, sql.ErrNoRows) {
        return nil, ErrNotFound
    }
    return &u, err
}

func (s *PostgresUserStore) ListUsers(ctx context.Context, limit, offset int) ([]*User, error) {
    var users []*User
    err := s.db.SelectContext(ctx, &users,
        "SELECT * FROM users ORDER BY created_at DESC LIMIT $1 OFFSET $2",
        limit, offset,
    )
    return users, err
}

func (s *PostgresUserStore) CreateUser(ctx context.Context, u *User) error {
    _, err := s.db.NamedExecContext(ctx,
        `INSERT INTO users (id, email, name, created_at, updated_at)
         VALUES (:id, :email, :name, :created_at, :updated_at)`, u)
    return err
}
为了便于测试,在调用层定义存储接口,针对具体数据库驱动实现。该模式可使领域逻辑与数据库解耦。
go
// 用于测试的Store接口
type UserStore interface {
    GetUser(ctx context.Context, id string) (*User, error)
    ListUsers(ctx context.Context, limit, offset int) ([]*User, error)
    CreateUser(ctx context.Context, u *User) error
}

// sqlx实现
type PostgresUserStore struct {
    db *sqlx.DB
}

func NewPostgresUserStore(db *sqlx.DB) *PostgresUserStore {
    return &PostgresUserStore{db: db}
}

func (s *PostgresUserStore) GetUser(ctx context.Context, id string) (*User, error) {
    var u User
    err := s.db.GetContext(ctx, &u, "SELECT * FROM users WHERE id = $1", id)
    if errors.Is(err, sql.ErrNoRows) {
        return nil, ErrNotFound
    }
    return &u, err
}

func (s *PostgresUserStore) ListUsers(ctx context.Context, limit, offset int) ([]*User, error) {
    var users []*User
    err := s.db.SelectContext(ctx, &users,
        "SELECT * FROM users ORDER BY created_at DESC LIMIT $1 OFFSET $2",
        limit, offset,
    )
    return users, err
}

func (s *PostgresUserStore) CreateUser(ctx context.Context, u *User) error {
    _, err := s.db.NamedExecContext(ctx,
        `INSERT INTO users (id, email, name, created_at, updated_at)
         VALUES (:id, :email, :name, :created_at, :updated_at)`, u)
    return err
}

Model Struct Tags

模型结构体标签

Use
db
tags for sqlx column mapping and keep models close to the store:
go
type User struct {
    ID        string    `db:"id"`
    Email     string    `db:"email"`
    Name      string    `db:"name"`
    CreatedAt time.Time `db:"created_at"`
    UpdatedAt time.Time `db:"updated_at"`
}
使用
db
标签实现sqlx列映射,并将模型与存储层放在一起:
go
type User struct {
    ID        string    `db:"id"`
    Email     string    `db:"email"`
    Name      string    `db:"name"`
    CreatedAt time.Time `db:"created_at"`
    UpdatedAt time.Time `db:"updated_at"`
}

Sentinel Errors

哨兵错误

Define domain-specific errors that callers can check without importing database packages:
go
var (
    ErrNotFound  = errors.New("not found")
    ErrConflict  = errors.New("conflict")
)
Map database errors to domain errors in the store layer:
go
func (s *PostgresUserStore) CreateUser(ctx context.Context, u *User) error {
    _, err := s.db.NamedExecContext(ctx, query, u)
    if err != nil {
        var pgErr *pgconn.PgError
        if errors.As(err, &pgErr) && pgErr.Code == "23505" {
            return ErrConflict
        }
        return fmt.Errorf("inserting user: %w", err)
    }
    return nil
}
定义领域专属错误,调用方无需导入数据库包即可检查:
go
var (
    ErrNotFound  = errors.New("not found")
    ErrConflict  = errors.New("conflict")
)
在存储层将数据库错误映射为领域错误:
go
func (s *PostgresUserStore) CreateUser(ctx context.Context, u *User) error {
    _, err := s.db.NamedExecContext(ctx, query, u)
    if err != nil {
        var pgErr *pgconn.PgError
        if errors.As(err, &pgErr) && pgErr.Code == "23505" {
            return ErrConflict
        }
        return fmt.Errorf("inserting user: %w", err)
    }
    return nil
}

Migrations

模式迁移

Use golang-migrate for managing schema changes. Migrations are pairs of SQL files: one for applying changes (
up
) and one for reverting them (
down
).
migrations/
├── 000001_create_users.up.sql
├── 000001_create_users.down.sql
├── 000002_add_user_roles.up.sql
└── 000002_add_user_roles.down.sql
Run migrations at application startup:
go
import "github.com/golang-migrate/migrate/v4"

func runMigrations(dbURL string) error {
    m, err := migrate.New("file://migrations", dbURL)
    if err != nil {
        return fmt.Errorf("creating migrator: %w", err)
    }

    if err := m.Up(); err != nil && err != migrate.ErrNoChange {
        return fmt.Errorf("running migrations: %w", err)
    }

    return nil
}
Key rules: always write both up and down migrations, use
IF NOT EXISTS
/
IF EXISTS
for idempotency, never modify a migration that has been applied in production. For full migration patterns, CI/CD integration, and safe migration strategies, see references/migrations.md.
使用golang-migrate管理模式变更。迁移文件为成对的SQL文件:一个用于应用变更(
up
),一个用于回滚变更(
down
)。
migrations/
├── 000001_create_users.up.sql
├── 000001_create_users.down.sql
├── 000002_add_user_roles.up.sql
└── 000002_add_user_roles.down.sql
在应用启动时执行迁移:
go
import "github.com/golang-migrate/migrate/v4"

func runMigrations(dbURL string) error {
    m, err := migrate.New("file://migrations", dbURL)
    if err != nil {
        return fmt.Errorf("creating migrator: %w", err)
    }

    if err := m.Up(); err != nil && err != migrate.ErrNoChange {
        return fmt.Errorf("running migrations: %w", err)
    }

    return nil
}
核心规则:始终同时编写up和down迁移文件,使用
IF NOT EXISTS
/
IF EXISTS
保证幂等性,绝不修改已在生产环境执行过的迁移文件。如需了解完整的迁移模式、CI/CD集成和安全迁移策略,请查看references/migrations.md

Transactions

事务管理

Use a transaction helper to ensure consistent commit/rollback handling. Transactions should be managed at the service layer, not the store layer, so that multiple store operations can be composed into a single atomic unit.
go
func WithTx(ctx context.Context, db *sql.DB, fn func(tx *sql.Tx) error) error {
    tx, err := db.BeginTx(ctx, nil)
    if err != nil {
        return fmt.Errorf("beginning transaction: %w", err)
    }

    if err := fn(tx); err != nil {
        if rbErr := tx.Rollback(); rbErr != nil {
            return fmt.Errorf("rollback failed: %v (original error: %w)", rbErr, err)
        }
        return err
    }

    return tx.Commit()
}
Store methods accept a
*sql.Tx
parameter so they can participate in a caller-controlled transaction:
go
func (s *OrderService) PlaceOrder(ctx context.Context, order *Order) error {
    return WithTx(ctx, s.db, func(tx *sql.Tx) error {
        if err := s.orderStore.CreateWithTx(ctx, tx, order); err != nil {
            return fmt.Errorf("creating order: %w", err)
        }
        if err := s.inventoryStore.DecrementWithTx(ctx, tx, order.Items); err != nil {
            return fmt.Errorf("updating inventory: %w", err)
        }
        return nil
    })
}
For isolation levels, deadlock prevention, context propagation, and testing strategies, see references/transactions.md.
使用事务工具确保提交/回滚的一致性处理。事务应在服务层管理,而非存储层,这样多个存储操作可组合为单个原子单元。
go
func WithTx(ctx context.Context, db *sql.DB, fn func(tx *sql.Tx) error) error {
    tx, err := db.BeginTx(ctx, nil)
    if err != nil {
        return fmt.Errorf("beginning transaction: %w", err)
    }

    if err := fn(tx); err != nil {
        if rbErr := tx.Rollback(); rbErr != nil {
            return fmt.Errorf("rollback failed: %v (original error: %w)", rbErr, err)
        }
        return err
    }

    return tx.Commit()
}
存储方法接受
*sql.Tx
参数,以便参与调用方控制的事务:
go
func (s *OrderService) PlaceOrder(ctx context.Context, order *Order) error {
    return WithTx(ctx, s.db, func(tx *sql.Tx) error {
        if err := s.orderStore.CreateWithTx(ctx, tx, order); err != nil {
            return fmt.Errorf("creating order: %w", err)
        }
        if err := s.inventoryStore.DecrementWithTx(ctx, tx, order.Items); err != nil {
            return fmt.Errorf("updating inventory: %w", err)
        }
        return nil
    })
}
如需了解隔离级别、死锁预防、上下文传递和测试策略,请查看references/transactions.md

When to Load References

何时查看参考文档

Load connection-pooling.md when:
  • Configuring pgx native pools (
    pgxpool.Pool
    )
  • Sizing connection pools for production workloads
  • Working with cloud databases, PgBouncer, or connection limits
  • Monitoring pool health and metrics
Load migrations.md when:
  • Setting up golang-migrate for the first time
  • Writing new migration files
  • Integrating migrations into CI/CD pipelines
  • Dealing with migration failures or rollbacks
Load transactions.md when:
  • Implementing multi-step operations that must be atomic
  • Designing service-layer transaction boundaries
  • Choosing transaction isolation levels
  • Debugging deadlocks or long-running transactions
查看connection-pooling.md的场景:
  • 配置pgx原生连接池(
    pgxpool.Pool
  • 为生产工作负载配置连接池大小
  • 使用云数据库、PgBouncer或处理连接限制
  • 监控连接池健康状态和指标
查看migrations.md的场景:
  • 首次配置golang-migrate
  • 编写新的迁移文件
  • 将迁移集成到CI/CD流水线
  • 处理迁移失败或回滚
查看transactions.md的场景:
  • 实现必须原子化的多步操作
  • 设计服务层事务边界
  • 选择事务隔离级别
  • 调试死锁或长运行事务

Anti-Patterns

反模式

Using string concatenation for queries

使用字符串拼接构建查询

go
// BAD -- SQL injection vulnerability
query := "SELECT * FROM users WHERE name = '" + name + "'"
Always use parameterized queries (
$1
,
$2
, etc.) or named parameters (
:name
).
go
// 错误做法 -- 存在SQL注入风险
query := "SELECT * FROM users WHERE name = '" + name + "'"
始终使用参数化查询(
$1
$2
等)或命名参数(
:name
)。

Leaking database types into handlers

向处理层暴露数据库类型

go
// BAD -- handler depends on sql.ErrNoRows
func (s *Server) handleGetUser(w http.ResponseWriter, r *http.Request) {
    user, err := s.store.GetUser(ctx, id)
    if errors.Is(err, sql.ErrNoRows) { // handler knows about sql package
        http.NotFound(w, r)
        return
    }
}
Return domain errors (
ErrNotFound
) from the store and check those in handlers instead.
go
// 错误做法 -- 处理层依赖sql.ErrNoRows
func (s *Server) handleGetUser(w http.ResponseWriter, r *http.Request) {
    user, err := s.store.GetUser(ctx, id)
    if errors.Is(err, sql.ErrNoRows) { // 处理层感知sql包
        http.NotFound(w, r)
        return
    }
}
存储层应返回领域错误(
ErrNotFound
),处理层检查这些领域错误而非数据库原生错误。

Opening a new connection per request

每次请求创建新连接

go
// BAD -- bypasses connection pooling entirely
func (s *Server) handleGetUser(w http.ResponseWriter, r *http.Request) {
    db, _ := sql.Open("postgres", connStr) // new pool per request!
    defer db.Close()
}
Open the database connection once at startup and share the pool across the application.
go
// 错误做法 -- 完全绕过连接池
func (s *Server) handleGetUser(w http.ResponseWriter, r *http.Request) {
    db, _ := sql.Open("postgres", connStr) // 每次请求创建新连接池!
    defer db.Close()
}
应在应用启动时创建一次数据库连接池,并在整个应用中共享。

SELECT * in production code

生产代码中使用SELECT *

go
// BAD -- fragile, breaks when columns change
err := db.GetContext(ctx, &u, "SELECT * FROM users WHERE id = $1", id)
Explicitly list the columns you need. This makes the query resilient to schema changes and avoids fetching unnecessary data.
go
// 错误做法 -- 脆弱,列变更时会失效
err := db.GetContext(ctx, &u, "SELECT * FROM users WHERE id = $1", id)
应显式列出所需列。这能使查询不受模式变更影响,避免获取不必要的数据。

Not handling context cancellation

不处理上下文取消

go
// BAD -- ignores context, query runs even if client disconnects
rows, err := db.Query("SELECT * FROM large_table")
Always use the
Context
variants (
QueryContext
,
ExecContext
,
GetContext
) and pass the request context so that queries are cancelled when the caller gives up.
go
// 错误做法 -- 忽略上下文,客户端断开后查询仍会执行
rows, err := db.Query("SELECT * FROM large_table")
始终使用带Context的变体方法(
QueryContext
ExecContext
GetContext
),并传递请求上下文,这样当调用方放弃时查询会被取消。

Transactions in store methods

在存储层管理事务

go
// BAD -- store controls transaction, caller cannot compose
func (s *UserStore) CreateUser(ctx context.Context, u *User) error {
    tx, _ := s.db.BeginTx(ctx, nil)
    // ... insert user ...
    return tx.Commit()
}
Let the service layer manage transactions and pass
*sql.Tx
into store methods. See references/transactions.md for the correct pattern.
go
// 错误做法 -- 存储层控制事务,调用方无法组合操作
func (s *UserStore) CreateUser(ctx context.Context, u *User) error {
    tx, _ := s.db.BeginTx(ctx, nil)
    // ... 插入用户 ...
    return tx.Commit()
}
应由服务层管理事务,将
*sql.Tx
传入存储方法。正确模式请查看references/transactions.md