blob-store

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Blob store

二进制大对象存储（Blob store）

Store large, immutable, unstructured objects — images, video, backups, model weights, document blobs — in a flat namespace keyed by a string, replicated for durability and served by direct download. Getting it wrong means stuffing multi-megabyte blobs into a row-oriented database (where they bloat the working set, wreck cache locality, and cap throughput) or hand-rolling a file server that loses data on the first disk failure.

存储大型、不可变的非结构化对象——图片、视频、备份、模型权重、文档二进制块——采用以字符串为键的扁平命名空间，通过复制保证持久性，并支持直接下载。如果使用不当，会导致将多兆字节的二进制块塞入面向行的数据库（这会膨胀工作集、破坏缓存局部性并限制吞吐量），或者手动搭建文件服务器，在第一次磁盘故障时就丢失数据。

When to reach for this

适用场景

Objects are large (KB to GB), written once and read many times, and you only ever fetch them whole by key — never query inside them. Photo/video stores, user uploads, backups, data-lake/ML datasets, static-site assets, log archives. The access pattern is

PUT key → GET key

, durability matters, and the total volume is too large or too cold to sit in a primary database.

对象体积较大（KB到GB级别），写入一次、读取多次，且仅通过键完整获取对象——从不查询对象内部内容。例如图片/视频存储、用户上传文件、备份、数据湖/机器学习数据集、静态站点资源、日志归档。访问模式为

PUT key → GET key

，数据持久性至关重要，且总数据量过大或访问频率过低，不适合存放在主数据库中。

When NOT to

不适用场景

Small structured records you query, filter, sort, or join — that is

data-storage

. Data that needs transactions, secondary indexes, or partial updates (blobs are replace-whole, not edit-in-place). Low-latency reads of tiny values (a KV cache or

caching

wins). A few files on one box that never grow — the local filesystem is fine; a blob store is operational overhead you do not need yet (YAGNI). Naming "object storage" for a workload that is really a database is failure mode #2.

需要查询、过滤、排序或关联的小型结构化记录——这类场景应使用

data-storage

。需要事务、二级索引或部分更新的数据（二进制对象只能整体替换，无法原地编辑）。对微小数据的低延迟读取（KV缓存或

caching

更合适）。单台机器上少量且不会增长的文件——本地文件系统足够；此时使用二进制对象存储会带来不必要的运维开销（YAGNI原则）。将实际应为数据库的工作负载命名为「对象存储」是第二种错误模式。

Clarify first

需先明确的问题

Object size distribution — average and p99 size? (Decides chunking, multipart thresholds, and whether reads stream or buffer.) →
```
back-of-the-envelope
```
.
Read:write ratio and access recency — write-once/read-many? How fast does data go cold? (Drives tiering and CDN fronting.)
Durability and availability target — how many nines of durability? Can a read briefly fail or must it always succeed? (Replication vs erasure coding, multi-region.)
Access control — public, private, or time-limited per-object grants? (Signed URLs.)
Mutability and history — do objects change? Must old versions be retained (compliance, undo)? (Versioning + lifecycle.)
Egress profile — who reads, from where, how often? (CDN offload, egress cost.)

对象大小分布——平均大小和99分位大小？（决定分块策略、分段上传阈值，以及读取是流式还是缓冲式。）→
```
back-of-the-envelope
```
。
读写比与访问时效性——是否是一次写入多次读取？数据多久会变为冷数据？（决定存储分层和CDN前置策略。）
持久性与可用性目标——需要几个9的持久性？读取是否允许短暂失败，还是必须始终成功？（复制 vs 纠删码、多区域部署。）
访问控制——公开、私有，还是针对单个对象的限时授权？（签名URL。）
可变性与历史记录——对象是否会变更？是否需要保留旧版本（合规、撤销操作）？（版本控制 + 生命周期管理。）
流出流量概况——谁读取数据、从哪里读取、读取频率如何？（CDN分流、流出成本。）

The options

可选方案

Durability scheme (how many copies, what shape)

N-way replication — store N full copies on different nodes/racks/AZs. Use when objects are small, hot, and latency matters; simplest to reason about.
Erasure coding (EC) — split an object into k data + m parity shards; any k reconstruct it. Use for large/cold data at scale — same durability as replication at ~1.4x overhead instead of 3x. (Mechanics in
```
references/deep-dive.md
```
.)

Storage tier (price/latency/retrieval trade)

Hot/standard — millisecond reads, highest $/GB. Use for actively served objects.
Cool/infrequent — cheaper storage, retrieval fee/slightly higher latency. Use for backups and data read a few times a month.
Archive/cold — cheapest storage, minutes-to-hours retrieval. Use for compliance retention and rarely-touched data; never for anything on a request path.

Upload path

Single PUT — one request. Use for small objects (under the multipart threshold).
Multipart / resumable — split into parts, upload in parallel, retry per-part, commit on completion. Use for large objects and flaky networks; the default above the threshold.

Mutation model

Immutable + versioning — each write is a new version; deletes are tombstones. Use when history, undo, or accidental-overwrite protection matters.
Overwrite-in-place (last-writer-wins) — simplest; no history. Use when only the latest object matters and storage of old copies is waste.

持久性方案（副本数量及形式）

N副本复制——在不同节点/机架/可用区存储N份完整副本。适用于对象体积小、访问频繁（热数据）且延迟敏感的场景；逻辑最简单，易于理解。
纠删码（Erasure coding, EC）——将对象拆分为k个数据分片 + m个校验分片；任意k个分片即可重构对象。适用于大规模的大型/冷数据——与复制方案具有相同的持久性，但存储开销仅约1.4倍，而非3倍。（详细机制见
```
references/deep-dive.md
```
。）

存储分层（价格/延迟/检索的权衡）

热/标准层——毫秒级读取速度，每GB成本最高。适用于活跃访问的对象。
冷/低频访问层——存储成本更低，但有检索费用且延迟略高。适用于备份和每月仅读取几次的数据。
归档/极冷层——存储成本最低，检索需要数分钟到数小时。适用于合规性留存和极少访问的数据；绝不能用于请求路径中的任何场景。

上传方式

单次PUT请求——一个请求完成上传。适用于小型对象（低于分段上传阈值）。
分段/可恢复上传——将对象拆分为多个部分，并行上传，可重试单个部分，完成后提交。适用于大型对象和不稳定网络；超过阈值时默认使用此方式。

变更模型

不可变 + 版本控制——每次写入都是一个新版本；删除操作仅标记墓碑。适用于需要历史记录、撤销操作或防止意外覆盖的场景。
原地覆盖（最后写入者获胜）——最简单的模式；无历史记录。适用于仅关心最新对象、存储旧副本属于浪费的场景。

Trade-offs

权衡对比

Option	What it solves	What it worsens	Change it when
N-way replication	Simple, fast reads, fast rebuild	3x+ storage cost	Data is large/cold and cost dominates → erasure coding
Erasure coding	Same durability at ~1.4x storage	CPU + multi-node read on every fetch; slow small-object reads; costly rebuild	Objects are small/hot and latency matters → replication
Hot tier	Low-latency serving	Highest $/GB	Data goes cold and is rarely read → cool/archive
Archive tier	Cheapest at-rest storage	Minutes–hours to first byte; retrieval fees	Anything ends up on a latency-sensitive path → hot/cool
Multipart/resumable upload	Large files survive flaky links; parallel throughput	More client logic; orphaned parts cost money	Objects are small → single PUT
Versioning	Undo, history, overwrite protection	Storage grows silently; needs lifecycle expiry	Only latest matters → overwrite, last-writer-wins
Signed URLs	Offload transfer off your app; scoped access	Leaked/over-broad URLs; clock-skew expiry bugs	Content is fully public → CDN + public read

选项	解决的问题	带来的问题	何时更换
N副本复制	逻辑简单、读取速度快、重建速度快	存储成本达3倍以上	数据为大型/冷数据且成本成为主导因素 → 切换为纠删码
纠删码	相同持久性下存储开销仅约1.4倍	每次读取都需要CPU计算 + 多节点读取；小型对象读取速度慢；重建成本高	对象为小型/热数据且延迟敏感 → 切换为复制
热层	低延迟服务	每GB成本最高	数据变为冷数据且极少被读取 → 切换为冷/归档层
归档层	最低的静态存储成本	首字节响应延迟为数分钟到数小时；有检索费用	数据进入延迟敏感的请求路径 → 切换为热/冷层
分段/可恢复上传	大型文件可在不稳定网络下完成上传；并行吞吐量高	需要更多客户端逻辑；未完成的分片会产生隐性成本	对象体积小 → 切换为单次PUT请求
版本控制	支持撤销操作、保留历史记录、防止意外覆盖	存储量会悄然增长；需要生命周期过期策略	仅关心最新对象 → 切换为原地覆盖（最后写入者获胜）
签名URL	减轻应用服务器的传输负载；实现细粒度访问控制	URL泄露或权限过宽；时钟偏差导致过期错误	内容完全公开 → 切换为CDN + 公开读取

Behavior under stress

压力下的表现

A blob store rarely "falls over" the way a database does, but it amplifies trouble in specific ways.

Hot object / hot prefix: a viral file or a key scheme where many writes share a prefix concentrates load on one partition. Mitigate: front hot reads with a CDN (
```
content-delivery
```
), randomize/hash key prefixes, replicate the hot object.
Metadata-index bottleneck: the index that maps key → shard locations is the real SPOF and the throughput ceiling (millions of tiny objects hurt far more than a few huge ones). Mitigate: shard the index, cache hot lookups, prefer fewer-larger objects.
Thundering herd on cold-tier promotion: a burst of reads for archived data triggers slow, expensive bulk retrievals. Mitigate: expose retrieval as async, queue it (
```
messaging-streaming
```
), set expectations on latency.
Orphaned multipart uploads: aborted uploads leave parts that silently accrue cost. Mitigate: lifecycle rule to abort incomplete uploads after N days.
Egress storm / cost blow-up: a popular object served directly from origin saturates bandwidth and runs up egress bills. Mitigate: CDN in front; the origin should serve cache fills, not end users.
Partial failure during write: a node dies mid-write. Mitigate: write is not acknowledged until the durability quorum (replicas or EC shards) is met; background repair re-replicates under-durable objects.

Monitor: request rate and error rate per operation (PUT/GET/DELETE), p99 first-byte latency, durability/repair queue depth, per-prefix hotness, incomplete-multipart count, and egress volume + cost.

二进制对象存储很少像数据库那样“崩溃”，但会以特定方式放大问题。

热门对象/热门前缀： viral文件或键命名方案导致大量写入共享同一前缀，会将负载集中到单个分区。缓解方案：用CDN前置热门读取请求（
```
content-delivery
```
），随机化/哈希键前缀，复制热门对象。
元数据索引瓶颈：映射键到分片位置的索引是真正的单点故障（SPOF）和吞吐量上限（数百万个小型对象比少数大型对象的危害大得多）。缓解方案：分片索引，缓存热门查询，优先选择数量少、体积大的对象。
冷层升级时的突发流量：对归档数据的突发读取会触发缓慢且昂贵的批量检索。缓解方案：将检索暴露为异步操作，加入队列（
```
messaging-streaming
```
），明确告知用户延迟预期。
未完成的分段上传：中断的上传会留下分片，悄然产生成本。缓解方案：设置生命周期规则，N天后终止未完成的上传。
流出流量风暴/成本激增：直接从源站提供热门对象会耗尽带宽并导致流出费用飙升。缓解方案：前置CDN；源站仅处理缓存填充请求，而非直接服务终端用户。
写入期间的部分故障：节点在写入过程中宕机。缓解方案：只有当满足持久性法定人数（副本或纠删码分片）时才确认写入；后台修复进程重新复制持久性不足的对象。

监控指标：各操作（PUT/GET/DELETE）的请求速率和错误率，99分位首字节延迟，持久性/修复队列深度，前缀热度，未完成分段上传数量，以及流出流量体积 + 成本。

How to apply

实施步骤

Clarify the inputs — pin object size distribution, read:write ratio, durability target, access control, and egress profile (see Clarify first). If the data is really small structured records you query, stop — use
```
data-storage
```
.
Pick from the trade-off table — choose a durability scheme (replication for small/hot, erasure coding for large/cold), a default tier, an upload path keyed to size, and a mutation model keyed to whether history matters.
Set the key knobs — design the key/prefix scheme to avoid hotspots, set the multipart threshold and part size, define lifecycle rules (tier transitions, version expiry, abort-incomplete-uploads), and decide signed-URL TTLs.
Stress-test the choice — walk each item in Behavior under stress (hot object, metadata bottleneck, cold-retrieval herd, orphaned parts, egress storm) and confirm a mitigation exists for the ones this traffic can trigger.
Size it with numbers — total storage with replication/EC overhead, object count (metadata-index load), peak PUT/GET QPS, and monthly egress. → Numbers that matter.
Pick a provider — default to the generic recipe; open a provider file only if the user named a cloud (see Choosing a provider).

明确输入条件——确定对象大小分布、读写比、持久性目标、访问控制和流出流量概况（见「需先明确的问题」）。如果数据是需要查询的小型结构化记录，请停止——使用
```
data-storage
```
。
根据权衡表选择方案——选择持久性方案（小型/热数据用复制，大型/冷数据用纠删码）、默认存储层、与对象大小匹配的上传方式，以及是否需要历史记录对应的变更模型。
设置关键参数——设计键/前缀方案以避免热点，设置分段上传阈值和分片大小，定义生命周期规则（分层转换、版本过期、终止未完成上传），并确定签名URL的TTL。
压力测试所选方案——逐一检查「压力下的表现」中的每个场景（热门对象、元数据瓶颈、冷检索突发流量、未完成分片、流出流量风暴），确认针对当前流量可能触发的场景有缓解方案。
量化规模——计算包含复制/纠删码开销的总存储量、对象数量（元数据索引负载）、峰值PUT/GET QPS，以及月度流出流量。→ Numbers that matter。
选择服务商——默认使用上述通用方案；仅当用户指定云服务商时，才查看
```
references/providers/<provider>.md
```
获取托管服务映射、配额/限制和服务商特定的权衡点。如果该服务商没有对应的文件，则通用方案即为答案。

Dos and don'ts

注意事项

Keep the blob in the store and a pointer (key + metadata) in the database — never the bytes in a row.
Treat objects as immutable; version instead of editing in place when history matters.
Front public/hot reads with a CDN so the origin serves fills, not end users.
Use multipart/resumable upload above the size threshold, and abort incomplete uploads via lifecycle.
Design key prefixes to spread load; hash or shuffle high-cardinality prefixes.
Set lifecycle rules to move cold data down tiers and expire old versions automatically.

Don't

Don't store large blobs in a relational/NoSQL row — it wrecks the working set and throughput.
Don't put archive tiers on a read path; minutes-to-hours retrieval will time out requests.
Don't serve a popular object directly from origin without a CDN (egress blowup).
Don't hand out broad or long-lived signed URLs; scope them tight and short.
Don't assume erasure coding is free — it adds CPU, multi-node reads, and expensive rebuilds.
Don't ignore millions-of-tiny-objects: the metadata index, not the disks, is the ceiling.

建议

将二进制对象保存在存储服务中，在数据库中存储指针（键 + 元数据）——绝不要将字节数据存在数据库行中。
将对象视为不可变；当需要历史记录时，使用版本控制而非原地编辑。
为公开/热门读取请求前置CDN，让源站仅处理缓存填充请求，而非直接服务终端用户。
超过大小阈值时使用分段/可恢复上传，并通过生命周期规则终止未完成的上传。
设计键前缀以分散负载；对高基数前缀进行哈希或随机化处理。
设置生命周期规则，自动将冷数据移至更低层级的存储，并过期旧版本。

禁忌

不要在关系型/NoSQL数据库的行中存储大型二进制对象——这会破坏工作集和吞吐量。
不要将归档层用于读取路径；数分钟到数小时的检索延迟会导致请求超时。
不要在没有CDN的情况下直接从源站提供热门对象（会导致流出成本激增）。
不要发放权限过宽或有效期过长的签名URL；应严格限定权限和有效期。
不要认为纠删码是免费的——它会增加CPU开销、多节点读取成本和昂贵的重建费用。
不要忽视数百万个小型对象的影响：元数据索引而非磁盘才是性能瓶颈。

Numbers that matter

关键量化指标

The figures that drive the design: total stored bytes × durability overhead (replication ≈ 3x, erasure coding ≈ 1.3–1.5x), object count (the metadata index scales with count, not size), peak upload/download QPS, and monthly egress (often the dominant cost). Object-store durability targets are commonly quoted around eleven nines; treat that as a design goal set by the durability scheme, not a given. For the actual storage/bandwidth/QPS arithmetic and unit conversions, use

back-of-the-envelope

— do not restate its tables here.

驱动设计的关键数据：总存储字节数 × 持久性开销（复制≈3倍，纠删码≈1.3–1.5倍）、对象数量（元数据索引的扩展与数量而非大小相关）、峰值上传/下载QPS，以及月度流出流量（通常是主要成本）。对象存储的持久性目标通常为11个9；这应视为由持久性方案设定的设计目标，而非既定事实。对于实际的存储/带宽/QPS计算和单位转换，请使用

back-of-the-envelope

——不要在此重复其表格内容。

Interface sketch

接口示例

The contract is small and key-addressed:

PUT    /{bucket}/{key}        body=bytes, headers: Content-Type, optional checksum
GET    /{bucket}/{key}        → bytes (supports Range for partial/streamed reads)
DELETE /{bucket}/{key}        → tombstone (new version if versioning on)
HEAD   /{bucket}/{key}        → metadata only (size, etag, version-id)

契约简洁，基于键寻址：

PUT    /{bucket}/{key}        body=bytes, headers: Content-Type, optional checksum
GET    /{bucket}/{key}        → bytes (supports Range for partial/streamed reads)
DELETE /{bucket}/{key}        → tombstone (new version if versioning on)
HEAD   /{bucket}/{key}        → metadata only (size, etag, version-id)

multipart: Initiate → UploadPart×N (parallel, retryable) → Complete | Abort

signed URL: presign(GET|PUT, key, expiry) → time-limited URL the client uses directly

The **key** is the whole index (e.g. `userId/2026/photo-uuid.jpg`); choose it for both
access pattern and prefix spread. The **etag/checksum** lets clients verify integrity and
do conditional requests. The database stores this key plus app metadata, not the bytes.

**键**是完整的索引（例如`userId/2026/photo-uuid.jpg`）；选择键时需兼顾访问模式和前缀分散性。**etag/校验和**让客户端可以验证完整性并执行条件请求。数据库存储此键及应用元数据，而非字节数据。

Choosing a provider

选择服务商

Default to the generic recipe above. If the user names a cloud, read

references/providers/<provider>.md

for the managed-service mapping, quotas/limits, and provider-specific trade-offs. If no file exists for that provider, the generic recipe is the answer.

默认使用上述通用方案。如果用户指定云服务商，请阅读

references/providers/<provider>.md

获取托管服务映射、影响决策的限制，以及特定环境下的陷阱。如果该服务商没有对应的文件，则通用方案即为答案。

Diagram

示意图

To visualize the upload/download path (client → signed URL → store; CDN fronting GET; index mapping key → shards) or the EC write fan-out, use the in-plugin

architecture-diagram

skill — show the metadata index as a distinct node and the CDN as the edge layer. Do not embed Mermaid here.

如需可视化上传/下载路径（客户端 → 签名URL → 存储服务；CDN前置GET请求；索引映射键到分片）或纠删码写入扩散流程，请使用插件内的

architecture-diagram

技能——将元数据索引显示为独立节点，CDN显示为边缘层。请勿在此嵌入Mermaid图。

Related building blocks

References

参考资料

references/deep-dive.md
— chunking, the metadata index, erasure-coding math and read path, durability/repair, versioning + lifecycle, multipart internals, signed-URL mechanics. Read when designing the store in detail.
references/providers/{generic,aws,azure,gcp}.md
— service mappings, decision-changing limits, and pitfalls per environment.

references/deep-dive.md
— 分块策略、元数据索引、纠删码数学原理和读取路径、持久性/修复机制、版本控制 + 生命周期管理、分段上传内部原理、签名URL机制。在详细设计存储服务时阅读。
references/providers/{generic,aws,azure,gcp}.md
— 服务映射、影响决策的限制，以及各环境下的陷阱。

blob-store

Original

Translation

Blob store

二进制大对象存储（Blob store）

When to reach for this

适用场景

When NOT to

不适用场景

Clarify first

需先明确的问题

The options

可选方案

Trade-offs

权衡对比

Behavior under stress

压力下的表现

How to apply

实施步骤

Dos and don'ts

注意事项

Numbers that matter

关键量化指标

Interface sketch

接口示例

multipart: Initiate → UploadPart×N (parallel, retryable) → Complete | Abort

multipart: Initiate → UploadPart×N (parallel, retryable) → Complete | Abort

signed URL: presign(GET|PUT, key, expiry) → time-limited URL the client uses directly

signed URL: presign(GET|PUT, key, expiry) → time-limited URL the client uses directly

Choosing a provider

选择服务商

Diagram

示意图

Related building blocks

相关构建模块

References

参考资料