data-storage

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Data Storage

数据存储

Choose where records live, how they are keyed and queried, and how the store grows past a single machine. Storage is the hardest layer to change later: a wrong data model or shard key calcifies into a scaling ceiling, and getting replication wrong silently serves stale or lost data.

决定记录的存储位置、键值设置与查询方式，以及存储系统如何突破单节点实现扩容。存储层是后期最难修改的部分：错误的数据模型或分片键会固化为扩容瓶颈，而复制架构设计失误则会悄无声息地导致数据过期或丢失。

When to reach for this

适用场景

Any system that persists state: picking SQL vs NoSQL, designing a schema and its access paths, adding indexes, splitting a hot table, distributing data across nodes (sharding/partitioning), adding read replicas, or deciding what to denormalize. Reach here the moment "store the data" needs a concrete key and query shape.

任何需要持久化状态的系统：选择SQL vs NoSQL、设计模式与访问路径、添加索引、拆分热点表、跨节点分配数据（sharding/partitioning）、添加只读副本，或是决定反规范化的内容。当“存储数据”需要明确的键值与查询形态时，即可使用本技能。

When NOT to

不适用场景

Don't shard, add replicas, or reach for NoSQL before a number forces it (YAGNI). A single well-indexed relational node handles ~1k QPS and tens of GB to low TB comfortably — most systems never outgrow it. Sharding multiplies operational cost and breaks joins/transactions; add it only when one node's write throughput or dataset size is genuinely exceeded (→

back-of-the-envelope

). Caching reads (→

caching

) and adding read replicas are cheaper first moves than sharding.

在实际需求迫使你之前，不要进行分片、添加副本或选择NoSQL（YAGNI原则）。一个配置良好索引的关系型节点可以轻松处理约1k QPS以及数十GB到低TB级别的数据——大多数系统永远不会超出这个范围。分片会成倍增加运维成本，还会破坏关联查询/事务；只有当单节点的写入吞吐量或数据集大小确实超出极限时，才需要添加分片（参考

back-of-the-envelope

）。缓存读取（参考

caching

）和添加只读副本是比分片成本更低的优先方案。

Clarify first

先明确以下问题

Answer these before choosing a store or topology — they decide the design:

Data shape & relationships — flat key-value? rich relations needing joins? document blobs? a graph of connections? Drives SQL vs NoSQL.
Access patterns — how is data read and written, not just where it lives. Point lookups by key, range scans, ad-hoc queries, aggregations? Model the store around the queries it must serve.
Read:write ratio & scale — QPS each way, total size now and at retention. (→
```
back-of-the-envelope
```
for QPS, storage, and shard counts.)
Consistency need — must reads see the latest write, or is eventual OK? Are multi-record transactions required? (CAP/consistency theory →
```
consistency-coordination
```
.)
Latency & durability targets — p99 read/write budget, and how much recent data the system can afford to lose on a node failure.

在选择存储系统或拓扑结构之前，先回答这些问题——它们将决定设计方向：

数据形态与关系 —— 是扁平键值对？需要关联查询的复杂关系？文档型Blob？还是连接图？这会决定选择SQL还是NoSQL。
访问模式 —— 数据的读写方式，而不仅仅是存储位置。是通过键值进行点查询、范围扫描、临时查询还是聚合操作？围绕存储系统需要服务的查询来建模。
读写比例与规模 —— 双向的QPS、当前及留存期内的总数据量。（参考
```
back-of-the-envelope
```
计算QPS、存储量和分片数量。）
一致性需求 —— 读取必须看到最新写入的数据，还是最终一致性即可？是否需要多记录事务？（CAP/一致性理论参考
```
consistency-coordination
```
。）
延迟与持久性目标 —— p99读写延迟预算，以及节点故障时系统可以承受丢失多少近期数据。

The options

可选方案

Relational (SQL — Postgres, MySQL): strict schema, joins, ACID transactions. Use when data is relational, integrity matters, and queries are varied/ad-hoc — the safe default until a number rules it out.

Document (MongoDB, etc.): flexible schema, self-contained JSON-ish documents queried by structure. Use when records are read/written as a whole and the schema evolves; relationships are few.

Key-value (Redis, DynamoDB, Riak): O(1) get/put by key, no rich queries. Use when access is purely by a known key and massive throughput is needed.

Wide-column (Cassandra, Bigtable, HBase): rows keyed by partition, columns sparse, keys kept sorted for range scans. Use when writes are huge and the table can be designed around a few known query patterns.

Graph (Neo4j): nodes and edges. Use when the core queries traverse many-to-many relationships (social graph, recommendations).

Scaling moves (apply on top of any store):

Indexing — add a secondary structure so a query stops scanning. First lever.
Read replicas (leader-follower) — copy writes to followers that serve reads. Use when reads dominate and slight staleness is OK.
Federation — split DBs by function (users / products / forums). Use when functional domains scale independently and rarely join.
Sharding/partitioning — split one logical table across nodes by a shard key. Use when a single node's writes or dataset are exceeded. (This skill owns it.)
Denormalization — store redundant copies to skip joins. Use when reads vastly outnumber writes and joins are the bottleneck.

Polyglot persistence: use more than one of the above, each for what it's best at (e.g. Postgres for orders, Redis for sessions, a search index for full-text). The cost is operating and reconciling several stores.

关系型（SQL — Postgres、MySQL）： 严格模式、关联查询、ACID事务。适用场景：数据存在关联关系、完整性很重要、查询多样/临时——在实际数据量或需求排除它之前，这是安全的默认选择。

文档型（MongoDB等）： 灵活模式、可按结构查询的独立JSON类文档。适用场景：记录整体读写、模式会演进；关联关系较少。

键值型（Redis、DynamoDB、Riak）： 通过键值实现O(1)的读写操作，无复杂查询。适用场景：仅通过已知键值访问数据，且需要极高吞吐量。

宽列型（Cassandra、Bigtable、HBase）： 行按分区键存储，列稀疏，键值保持有序以支持范围扫描。适用场景：写入量极大，且表可以围绕少数已知查询模式设计。

图型（Neo4j）： 节点与边结构。适用场景：核心查询需要遍历多对多关系（社交图谱、推荐系统）。

扩容方案（可应用于任何存储系统）：

Indexing —— 添加二级结构，避免查询时全表扫描。这是首要手段。
只读副本（主从模式） —— 将写入操作复制到副本节点，由副本处理读取请求。适用场景：读取占主导，且可接受轻微的数据过期。
联邦化 —— 按功能拆分数据库（用户/产品/论坛）。适用场景：功能域独立扩容，且很少需要跨域关联查询。
Sharding/partitioning —— 按分片键将一个逻辑表拆分到多个节点。适用场景：单节点的写入或数据集超出极限。（本技能负责此方案。）
Denormalization —— 存储冗余数据以避免关联查询。适用场景：读取量远大于写入量，且关联查询是瓶颈。

Polyglot persistence： 同时使用上述多种方案，各取所长（例如用Postgres存储订单、Redis存储会话、搜索索引处理全文检索）。代价是需要运维和协调多个存储系统。

Trade-offs

权衡对比

Option	What it solves	What it worsens	Change it when
Relational/SQL	Joins, ACID, ad-hoc queries, integrity	Single-node write ceiling; schema migrations; harder horizontal scale	Writes/size exceed one node, or schema is truly fluid → NoSQL/shard
Document	Schema flexibility; whole-object reads	No joins; cross-document consistency is manual; query engine weaker	Data turns relational or needs multi-doc transactions → SQL
Key-value	Extreme throughput, simple ops	Only key access; no range/secondary queries	Queries beyond the key are needed → document/wide-column
Wide-column	Write-heavy scale, range scans on sorted keys	Must know queries up front; rigid once keyed; eventual by default	Access patterns are unknown/varied → relational
Graph	Cheap deep relationship traversal	Niche tooling; hard to shard; weak for bulk scans	Relationships are shallow → relational/document
Indexing	Turns scans into lookups	Slower writes; more storage; index bloat	Write amplification hurts more than the read win
Read replicas	Offloads reads; redundancy	Replication lag → stale reads; failover/promotion logic	Stale reads unacceptable → read-from-leader / `consistency-coordination`
Federation	Per-domain scale, smaller working sets	Cross-domain joins break; app routing logic	A single domain still won't fit → shard that domain
Sharding	Horizontal write + storage scale	Cross-shard joins/txns hard; resharding pain; hot shards	One node holds it fine, or hot shards dominate → consolidate/re-key
Denormalization	Kills expensive joins on the read path	Duplicated data; write-time fan-out; consistency drift	Write load makes fan-out the new bottleneck → normalize/cache

方案	解决的问题	带来的弊端	何时更换
关系型/SQL	关联查询、ACID、临时查询、数据完整性	单节点写入上限；模式迁移；水平扩容难度大	写入量/数据量超出单节点，或模式确实需要灵活变更 → NoSQL/分片
文档型	模式灵活性；整对象读写	无关联查询；跨文档一致性需手动维护；查询引擎较弱	数据变为关系型或需要多文档事务 → SQL
键值型	极高吞吐量、运维简单	仅支持键值访问；无范围/二级查询	需要键值之外的查询 → 文档型/宽列型
宽列型	写入密集型扩容、基于有序键的范围扫描	必须提前明确查询模式；键值设定后难以修改；默认最终一致性	访问模式未知/多变 → 关系型
图型	低成本深度关系遍历	工具生态小众；分片难度大；批量扫描性能弱	关系较浅 → 关系型/文档型
Indexing	将全表扫描转为点查询	写入速度变慢；占用更多存储；索引膨胀	写入放大的负面影响超过读取性能提升
只读副本	分流读取请求；冗余备份	复制延迟 → 读取过期数据；故障切换/提升逻辑复杂	无法接受过期数据 → 从主节点读取 / `consistency-coordination`
联邦化	按域扩容、减小工作集	跨域关联查询失效；应用路由逻辑复杂	单个域仍无法容纳 → 对该域进行分片
Sharding	水平写入+存储扩容	跨分片关联查询/事务难度大；重分片成本高；热点分片	单节点可容纳，或热点分片占主导 → 合并/重新设置键值
Denormalization	消除读取路径上的昂贵关联查询	数据重复；写入时需扩散数据；一致性漂移	写入负载使数据扩散成为新瓶颈 → 规范化/缓存

Behavior under stress

压力下的表现

Storage is where load and failure get amplified into outages.

Hot shard / celebrity key: a skewed shard key sends disproportionate traffic to one node (a viral user, a single tenant) while others idle. The cluster looks under-loaded but one shard is melting. Mitigate: a shard key with high cardinality and even distribution; isolate or sub-partition hot keys.
Replication lag: under write bursts, followers fall behind the leader, so reads go stale — and a failover may promote a follower that lost recent writes. Mitigate: read-from-leader for read-your-writes; bound and alert on lag.
Connection exhaustion: each DB connection costs memory and a backend process/thread; a traffic spike (or a retry storm) opens more connections than the DB can serve, and every query slows or errors. A pooler in front (bounded pool) is what keeps the DB alive — without it the database is a SPOF that fails under load.
Thundering writes / lock contention: hot rows or a single sequence/counter serialize writes; index updates and lock waits stack up.
Resharding under pressure: rebalancing while already overloaded moves huge data volumes and can tip the cluster over. Plan capacity ahead of the cliff.

Monitor: replication lag, per-shard QPS and size (skew), connection-pool saturation/wait time, slow-query rate, lock waits, and disk/IOPS headroom.

存储层是负载和故障会被放大为宕机的环节。

热点分片/明星键： 分片键分布不均导致某一节点接收不成比例的流量（如热门用户、单个租户），而其他节点闲置。集群整体看似负载不高，但单个分片已濒临崩溃。缓解方案：选择高基数且分布均匀的分片键；隔离或拆分热点键。
复制延迟： 写入突发时，副本节点落后于主节点，导致读取到过期数据——故障切换时可能会提升丢失近期写入数据的副本为主节点。缓解方案：读取自己写入的数据时从主节点读取；监控并限制延迟阈值。
连接耗尽： 每个数据库连接都会占用内存和后端进程/线程；流量峰值（或重试风暴）会打开超出数据库承载能力的连接数，导致所有查询变慢或报错。前端设置连接池（有界池）是维持数据库存活的关键——没有它，数据库会成为单点故障，在负载下崩溃。
突发写入/锁竞争： 热点行或单个序列/计数器会使写入操作串行化；索引更新和锁等待会堆积。
压力下重分片： 在已过载的情况下进行重平衡会移动大量数据，可能导致集群崩溃。提前规划容量，避免临近极限。

监控指标： 复制延迟、每分片QPS与数据量（分布不均）、连接池饱和度/等待时间、慢查询率、锁等待时间、磁盘/IOPS余量。

How to apply

实施步骤

Clarify the inputs — settle data shape, access patterns, read:write ratio, consistency need, and latency/durability targets (see Clarify first). No store choice survives unknown access patterns.
Pick the store from the trade-off table — match data shape to an option; default to relational until a number or relationship pattern rules it out. Name what each candidate worsens, not just what it solves.
Pin the interface — write the primary key, partition/sort key, and the secondary indexes for each query before adding scale machinery (see Interface sketch). The key is the decision.
Set the scaling knobs in cheap-first order — index, then cache (→
```
caching
```
), then read replicas, then federation, then shard. Stop at the first level that meets the target.
Stress-test the choice — walk hot shard, replication lag, connection exhaustion, and resharding (see Behavior under stress); confirm a skewed key or write burst does not melt one node.
Size it, then pick a provider — compute shard count and per-node load from
```
back-of-the-envelope
```
; if it is one node, do not shard. Default to the generic recipe and read the provider file only when a cloud is named.

明确输入条件 —— 确定数据形态、访问模式、读写比例、一致性需求以及延迟/持久性目标（参考先明确以下问题）。未知访问模式下的存储系统选择无法成立。
根据权衡表选择存储系统 —— 匹配数据形态与可选方案；在数据量或关系模式排除它之前，默认选择关系型。说明每个候选方案的弊端，而不仅仅是优势。
确定接口 —— 在添加扩容机制之前，为每个查询写入主键、分区/排序键以及二级索引（参考接口草图）。键值是核心决策。
按成本从低到高设置扩容选项 —— 先添加索引，再缓存（参考
```
caching
```
），然后添加只读副本，接着联邦化，最后分片。达到目标后即停止。
压力测试所选方案 —— 模拟热点分片、复制延迟、连接耗尽和重分片场景（参考压力下的表现）；确认分布不均的键值或写入突发不会导致单个节点崩溃。
估算规模，选择服务商 —— 通过
```
back-of-the-envelope
```
计算分片数量和单节点负载；如果只需单节点，则不要分片。默认使用通用方案，仅当指定云服务商时才查看对应服务商文档。

Dos and don'ts

注意事项

Start from the queries: model the store around its access paths, and write a concrete key before drawing any box.
Default to a single well-indexed relational node and exhaust index + cache + replicas before sharding.
Choose a shard key with high cardinality and even distribution; isolate or sub-partition known hot keys.
Put a bounded connection pooler in front of the database and alert on replication lag and per-shard skew.
Compute shard count from peak write QPS and dataset size, taking the larger.

Don't

Don't reach for NoSQL, replicas, or shards before a number forces it (YAGNI).
Don't pick a store on hype before the data shape and consistency need are known.
Don't assume replicas give fresh reads — replication lag serves stale data and failover can lose recent writes.
Don't shard on a low-cardinality or monotonically increasing key; it creates hot shards and write hotspots.
Don't reshard a cluster that is already overloaded; plan capacity ahead of the cliff.

建议

从查询出发：围绕访问路径建模存储系统，在绘制任何架构图之前先确定具体键值。
从单个配置良好索引的关系型节点开始，在分片之前先尝试索引+缓存+副本的组合。
选择高基数且分布均匀的分片键；隔离或拆分已知的热点键。
在数据库前端设置有界连接池，监控复制延迟和每分片分布不均情况并告警。
根据峰值写入QPS和数据集大小计算分片数量，取较大值。

禁忌

在实际需求迫使你之前，不要选择NoSQL、副本或分片（YAGNI原则）。
在明确数据形态和一致性需求之前，不要跟风选择存储系统。
不要假设副本能提供最新数据——复制延迟会导致读取过期数据，故障切换可能丢失近期写入数据。
不要选择低基数或单调递增的键值作为分片键；这会导致热点分片和写入热点。
不要在集群已过载时进行重分片；提前规划容量，避免临近极限。

Numbers that matter

关键数值

Don't restate the tables — pull the figures from

back-of-the-envelope

. The ones that drive storage decisions: a single RDBMS node ≈ 1k QPS; a key-value node ≈ 10k QPS; 10 GB fits in RAM, 10 TB needs distributed storage. Use these to compute shard count = peak write QPS ÷ per-node QPS (and again by size = total bytes ÷ per-node capacity), then take the larger. If the result is one node, do not shard.

不要重复表格内容——从

back-of-the-envelope

获取数据。驱动存储决策的关键数值：单个RDBMS节点≈1k QPS；单个键值型节点≈10k QPS；10 GB数据可放入内存，10 TB数据需要分布式存储。使用这些数值计算分片数量=峰值写入QPS÷单节点QPS（再按总字节数÷单节点容量计算一次），取较大值。如果结果为1，则不要分片。

Interface sketch

接口草图

The data model is the contract (GUIDE failure mode #8) — a "NoSQL box" decides nothing until the key is written. Pin down per entity:

Primary key — what uniquely identifies a row/item and how it's looked up.
Partition (shard) key + sort key — e.g. wide-column/DynamoDB:
```
PK = user_id
```
(hash, spreads load),
```
SK = created_at
```
(sort, enables range scans like "latest N posts"). The PK must be high-cardinality and even.
Secondary indexes — the non-key access paths that must be supported, each with its query.
Relational — tables, columns with types, foreign keys, and the indexes that back each query; note what is intentionally denormalized.

Without a concrete key and the query it serves, the rest of the design is guesswork.

数据模型就是契约（GUIDE故障模式#8）——在确定键值之前，“NoSQL架构”无法做出任何决策。为每个实体明确：

Primary key —— 唯一标识行/项的字段，以及查询方式。
Partition（shard）key + sort key —— 例如宽列型/DynamoDB：
```
PK = user_id
```
（哈希，分散负载），
```
SK = created_at
```
（排序，支持“最新N条帖子”等范围扫描）。PK必须具有高基数且分布均匀。
Secondary indexes —— 必须支持的非键访问路径，每个路径对应具体查询。
关系型 —— 表、带类型的列、外键，以及支持每个查询的索引；注明故意反规范化的内容。

如果没有具体的键值及其对应的查询，其余设计都只是猜测。

Choosing a provider

选择服务商

Default to the generic recipe above. If the user names a cloud, read

references/providers/<provider>.md

for the managed-service mapping, quotas/limits, and provider-specific trade-offs. If no file exists for that provider, the generic recipe is the answer.

默认使用上述通用方案。如果用户指定云服务商，查看

references/providers/<provider>.md

获取托管服务映射、配额/限制以及服务商特定的权衡点。如果该服务商没有对应文档，则使用通用方案。

Diagram

图表

To visualize the data tier — leader with read replicas, a sharded cluster with a router, or a polyglot split — use the in-plugin

architecture-diagram

skill. Draw the shard key on the routing arrow and the replication direction explicitly; do not embed Mermaid here.

要可视化数据层——带只读副本的主节点、带路由的分片集群或多存储系统拆分——使用插件内的

architecture-diagram

技能。在路由箭头上标注分片键，明确复制方向；不要在此处嵌入Mermaid代码。

Related building blocks

References

参考资料

references/deep-dive.md
— partitioning schemes (range/hash/directory), replication mechanics and conflict resolution, index internals (B-tree vs LSM), normalization vs denormalization, connection pooling, resharding. Read when designing the data tier in detail.
references/providers/{generic,aws,azure,gcp}.md
— service mappings, the limits that change a decision, and per-environment pitfalls.

references/deep-dive.md
—— 分区方案（范围/哈希/目录）、复制机制与冲突解决、索引内部原理（B树 vs LSM）、规范化与反规范化、连接池、重分片。详细设计数据层时阅读。
references/providers/{generic,aws,azure,gcp}.md
—— 服务映射、影响决策的限制、各环境的陷阱。