Back to Details

vector-database-ops

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

Vector Database Operations

向量数据库运维

Run production vector databases for AI-powered search, RAG, and recommendation systems.
为基于AI的搜索、RAG和推荐系统运行生产级向量数据库。

When to Use This Skill

何时使用此技能

Use this skill when:
  • Setting up a vector database for a RAG or semantic search application
  • Choosing between Qdrant, Weaviate, pgvector, or Pinecone
  • Managing collections, indexes, and data migrations
  • Optimizing query performance and indexing for production loads
  • Implementing multi-tenant vector search with namespace isolation
在以下场景使用此技能：
  • 为RAG或语义搜索应用搭建向量数据库
  • 在Qdrant、Weaviate、pgvector或Pinecone之间做选型
  • 管理集合、索引和数据迁移
  • 针对生产负载优化查询性能与索引
  • 实现带命名空间隔离的多租户向量搜索

Vector Database Comparison

向量数据库对比

DatabaseBest ForHostingFilteringScale
QdrantHigh-performance, rich filtering, self-hostedSelf / CloudExcellentVery High
WeaviateSchema-first, hybrid search, multi-modalSelf / CloudGoodHigh
pgvectorAlready on Postgres, simple use casesSelfGoodMedium
PineconeZero-ops managed, serverlessManaged onlyGoodVery High
ChromaLocal dev, prototypingSelf onlyBasicLow-Medium
数据库适用场景部署方式过滤能力扩展性
Qdrant高性能、丰富过滤能力、自托管自托管/云托管优秀极高
WeaviateSchema优先、混合搜索、多模态自托管/云托管良好高
pgvector已使用PostgreSQL、简单场景自托管良好中等
Pinecone零运维托管、无服务器仅托管良好极高
Chroma本地开发、原型验证仅自托管基础低-中等

Qdrant — Production Deployment

Qdrant — 生产环境部署

bash
undefined
bash
undefined

Docker (single node)

Docker (单节点)

docker run -d
--name qdrant
-p 6333:6333
-p 6334:6334
-v $(pwd)/qdrant-data:/qdrant/storage
qdrant/qdrant:latest
docker run -d
--name qdrant
-p 6333:6333
-p 6334:6334
-v $(pwd)/qdrant-data:/qdrant/storage
qdrant/qdrant:latest

With custom config

自定义配置启动

docker run -d
--name qdrant
-p 6333:6333
-v $(pwd)/qdrant-data:/qdrant/storage
-v $(pwd)/qdrant-config.yaml:/qdrant/config/production.yaml
qdrant/qdrant:latest

```yaml
docker run -d
--name qdrant
-p 6333:6333
-v $(pwd)/qdrant-data:/qdrant/storage
-v $(pwd)/qdrant-config.yaml:/qdrant/config/production.yaml
qdrant/qdrant:latest

```yaml

qdrant-config.yaml

qdrant-config.yaml

storage: storage_path: /qdrant/storage on_disk_payload: true # store payload on disk (saves RAM)
service: max_request_size_mb: 32
hnsw_index: m: 16 # graph connections per node ef_construct: 100 # accuracy vs build time trade-off full_scan_threshold: 10000 # switch to brute force below this
quantization: scalar: type: int8 quantile: 0.99 always_ram: true # keep quantized index in RAM
telemetry_disabled: true
undefined
storage: storage_path: /qdrant/storage on_disk_payload: true # 将负载存储在磁盘（节省内存）
service: max_request_size_mb: 32
hnsw_index: m: 16 # 每个节点的图连接数 ef_construct: 100 # 精度与构建时间的权衡 full_scan_threshold: 10000 # 低于此数量时切换为暴力搜索
quantization: scalar: type: int8 quantile: 0.99 always_ram: true # 将量化索引保留在内存中
telemetry_disabled: true
undefined

Qdrant Collection Management

Qdrant 集合管理

python
from qdrant_client import QdrantClient
from qdrant_client.models import (
    Distance, VectorParams, HnswConfigDiff,
    ScalarQuantizationConfig, ScalarType, QuantizationConfig
)

client = QdrantClient("http://localhost:6333")
python
from qdrant_client import QdrantClient
from qdrant_client.models import (
    Distance, VectorParams, HnswConfigDiff,
    ScalarQuantizationConfig, ScalarType, QuantizationConfig
)

client = QdrantClient("http://localhost:6333")

Create optimized collection

创建优化后的集合

client.create_collection( collection_name="documents", vectors_config=VectorParams( size=1536, # OpenAI ada-002 / text-embedding-3-small distance=Distance.COSINE, on_disk=True, # save RAM — vectors stored on disk ), hnsw_config=HnswConfigDiff( m=32, # higher = better recall, more RAM ef_construct=200, on_disk=False, # keep HNSW graph in RAM for speed ), quantization_config=QuantizationConfig( scalar=ScalarQuantizationConfig( type=ScalarType.INT8, quantile=0.99, always_ram=True, ) ), )
client.create_collection( collection_name="documents", vectors_config=VectorParams( size=1536, # OpenAI ada-002 / text-embedding-3-small distance=Distance.COSINE, on_disk=True, # 节省内存 — 向量存储在磁盘 ), hnsw_config=HnswConfigDiff( m=32, # 值越高召回率越好，占用内存越多 ef_construct=200, on_disk=False, # 将HNSW图保留在内存以提升速度 ), quantization_config=QuantizationConfig( scalar=ScalarQuantizationConfig( type=ScalarType.INT8, quantile=0.99, always_ram=True, ) ), )

Create payload index for fast filtering

创建负载索引以实现快速过滤

client.create_payload_index( collection_name="documents", field_name="tenant_id", field_schema="keyword", ) client.create_payload_index( collection_name="documents", field_name="created_at", field_schema="datetime", )
client.create_payload_index( collection_name="documents", field_name="tenant_id", field_schema="keyword", ) client.create_payload_index( collection_name="documents", field_name="created_at", field_schema="datetime", )

Collection info

查看集合信息

info = client.get_collection("documents") print(f"Vectors: {info.vectors_count}, Status: {info.status}")
undefined
info = client.get_collection("documents") print(f"向量数量: {info.vectors_count}, 状态: {info.status}")
undefined

Qdrant Filtered Search

Qdrant 过滤搜索

python
from qdrant_client.models import Filter, FieldCondition, MatchValue, Range
python
from qdrant_client.models import Filter, FieldCondition, MatchValue, Range

Tenant-isolated search (multi-tenant RAG)

租户隔离搜索（多租户RAG）

results = client.query_points( collection_name="documents", query=query_embedding, query_filter=Filter( must=[ FieldCondition(key="tenant_id", match=MatchValue(value="acme-corp")), FieldCondition(key="doc_type", match=MatchValue(value="contract")), ], should=[ FieldCondition(key="created_at", range=Range(gte="2024-01-01")), ], ), limit=10, with_payload=True, )
undefined
results = client.query_points( collection_name="documents", query=query_embedding, query_filter=Filter( must=[ FieldCondition(key="tenant_id", match=MatchValue(value="acme-corp")), FieldCondition(key="doc_type", match=MatchValue(value="contract")), ], should=[ FieldCondition(key="created_at", range=Range(gte="2024-01-01")), ], ), limit=10, with_payload=True, )
undefined

pgvector — PostgreSQL Extension

pgvector — PostgreSQL 扩展

sql
-- Enable extension
CREATE EXTENSION IF NOT EXISTS vector;

-- Create table with vector column
CREATE TABLE documents (
    id          UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    content     TEXT NOT NULL,
    embedding   VECTOR(1536),
    metadata    JSONB DEFAULT '{}',
    tenant_id   TEXT NOT NULL,
    created_at  TIMESTAMPTZ DEFAULT NOW()
);

-- Create HNSW index (faster queries, more memory)
CREATE INDEX ON documents
USING hnsw (embedding vector_cosine_ops)
WITH (m = 16, ef_construction = 64);

-- Create IVFFlat index (less memory, slower build)
-- CREATE INDEX ON documents
-- USING ivfflat (embedding vector_cosine_ops)
-- WITH (lists = 100);

-- Semantic search with metadata filtering
SELECT id, content, metadata,
       1 - (embedding <=> $1::vector) AS similarity
FROM documents
WHERE tenant_id = 'acme-corp'
  AND metadata->>'doc_type' = 'contract'
ORDER BY embedding <=> $1::vector
LIMIT 10;
bash
undefined
sql
-- 启用扩展
CREATE EXTENSION IF NOT EXISTS vector;

-- 创建带向量列的表
CREATE TABLE documents (
    id          UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    content     TEXT NOT NULL,
    embedding   VECTOR(1536),
    metadata    JSONB DEFAULT '{}',
    tenant_id   TEXT NOT NULL,
    created_at  TIMESTAMPTZ DEFAULT NOW()
);

-- 创建HNSW索引（查询更快，占用内存更多）
CREATE INDEX ON documents
USING hnsw (embedding vector_cosine_ops)
WITH (m = 16, ef_construction = 64);

-- 创建IVFFlat索引（占用内存更少，构建更慢）
-- CREATE INDEX ON documents
-- USING ivfflat (embedding vector_cosine_ops)
-- WITH (lists = 100);

-- 带元数据过滤的语义搜索
SELECT id, content, metadata,
       1 - (embedding <=> $1::vector) AS similarity
FROM documents
WHERE tenant_id = 'acme-corp'
  AND metadata->>'doc_type' = 'contract'
ORDER BY embedding <=> $1::vector
LIMIT 10;
bash
undefined

Deploy pgvector via Docker

通过Docker部署pgvector

docker run -d
--name pgvector
-e POSTGRES_PASSWORD=secret
-e POSTGRES_DB=vectordb
-p 5432:5432
-v pgvector-data:/var/lib/postgresql/data
pgvector/pgvector:pg16
undefined
docker run -d
--name pgvector
-e POSTGRES_PASSWORD=secret
-e POSTGRES_DB=vectordb
-p 5432:5432
-v pgvector-data:/var/lib/postgresql/data
pgvector/pgvector:pg16
undefined

Weaviate Deployment

Weaviate 部署

yaml
undefined
yaml
undefined

docker-compose for Weaviate

Weaviate的docker-compose配置

services: weaviate: image: semitechnologies/weaviate:latest ports: - "8080:8080" - "50051:50051" environment: QUERY_DEFAULTS_LIMIT: 25 AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: "false" AUTHENTICATION_APIKEY_ENABLED: "true" AUTHENTICATION_APIKEY_ALLOWED_KEYS: "${WEAVIATE_API_KEY}" AUTHENTICATION_APIKEY_USERS: "admin" PERSISTENCE_DATA_PATH: /var/lib/weaviate ENABLE_MODULES: text2vec-openai,generative-openai OPENAI_APIKEY: "${OPENAI_API_KEY}" CLUSTER_HOSTNAME: node1 volumes: - weaviate-data:/var/lib/weaviate restart: unless-stopped
volumes: weaviate-data:
undefined
services: weaviate: image: semitechnologies/weaviate:latest ports: - "8080:8080" - "50051:50051" environment: QUERY_DEFAULTS_LIMIT: 25 AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: "false" AUTHENTICATION_APIKEY_ENABLED: "true" AUTHENTICATION_APIKEY_ALLOWED_KEYS: "${WEAVIATE_API_KEY}" AUTHENTICATION_APIKEY_USERS: "admin" PERSISTENCE_DATA_PATH: /var/lib/weaviate ENABLE_MODULES: text2vec-openai,generative-openai OPENAI_APIKEY: "${OPENAI_API_KEY}" CLUSTER_HOSTNAME: node1 volumes: - weaviate-data:/var/lib/weaviate restart: unless-stopped
volumes: weaviate-data:
undefined

Backup and Restore

备份与恢复

bash
undefined
bash
undefined

Qdrant — snapshot backup

Qdrant — 快照备份

curl -X POST "http://localhost:6333/collections/documents/snapshots"
curl -X POST "http://localhost:6333/collections/documents/snapshots"

Download snapshot

下载快照

curl -O "http://localhost:6333/collections/documents/snapshots/documents-snapshot.snapshot"
curl -O "http://localhost:6333/collections/documents/snapshots/documents-snapshot.snapshot"

Restore

恢复快照

curl -X POST "http://localhost:6333/collections/documents/snapshots/recover"
-H "Content-Type: application/json"
-d '{"location": "/qdrant/snapshots/documents-snapshot.snapshot"}'
curl -X POST "http://localhost:6333/collections/documents/snapshots/recover"
-H "Content-Type: application/json"
-d '{"location": "/qdrant/snapshots/documents-snapshot.snapshot"}'

pgvector — standard pg_dump

pgvector — 使用标准pg_dump备份

pg_dump -h localhost -U postgres -d vectordb
--table=documents --format=custom > documents-backup.dump
pg_dump -h localhost -U postgres -d vectordb
--table=documents --format=custom > documents-backup.dump

Restore

恢复备份

pg_restore -h localhost -U postgres -d vectordb documents-backup.dump
undefined
pg_restore -h localhost -U postgres -d vectordb documents-backup.dump
undefined

Performance Tuning

性能调优

python
undefined
python
undefined

Qdrant — optimize collection after bulk load

Qdrant — 批量加载后优化集合

client.update_collection( collection_name="documents", optimizer_config={"indexing_threshold": 0}, # force indexing now )
client.update_collection( collection_name="documents", optimizer_config={"indexing_threshold": 0}, # 立即强制索引 )

Wait for optimization to complete

等待优化完成

import time while True: info = client.get_collection("documents") if info.status.value == "green": break time.sleep(5) print(f"Optimizing... segments: {info.segments_count}")
undefined
import time while True: info = client.get_collection("documents") if info.status.value == "green": break time.sleep(5) print(f"优化中... 段数: {info.segments_count}")
undefined

Common Issues

常见问题

IssueCauseFix
Slow queriesNo HNSW index built yetWait for indexing; check 
status == green
High RAM usageVectors in memoryEnable 
on_disk=True
for vectors
Poor recallLow 
ef
search param		Increase 
ef
in search request (at query time)
pgvector slowUsing IVFFlat without vacuumRun 
VACUUM ANALYZE documents
Weaviate OOMToo many objectsEnable async indexing; increase heap
问题原因解决方法
查询缓慢尚未构建HNSW索引等待索引完成；检查
status == green
内存占用过高向量存储在内存中启用
on_disk=True
将向量存储到磁盘
召回率低搜索参数
ef
设置过低		在查询时提高
ef
参数
pgvector查询缓慢使用IVFFlat但未执行vacuum运行
VACUUM ANALYZE documents
Weaviate内存溢出对象数量过多启用异步索引；增加堆内存

Best Practices

最佳实践

  • Use cosine distance for normalized embeddings; dot product for unnormalized.
  • Always create payload indexes on filter fields (
    tenant_id
    , 
    doc_type
    ).
  • For datasets >10M vectors, use 
    on_disk
    vectors + 
    always_ram
    quantization.
  • Benchmark with your actual query patterns before choosing IVFFlat vs HNSW.
  • Snapshot before any bulk delete or migration operation.
  • 归一化嵌入使用余弦距离；非归一化嵌入使用点积。
  • 务必为过滤字段（
    tenant_id
    、
    doc_type
    ）创建负载索引。
  • 对于超过1000万条向量的数据集，使用
    on_disk
    向量 + 
    always_ram
    量化。
  • 在选择IVFFlat与HNSW之前，用实际查询模式做基准测试。
  • 在执行批量删除或迁移操作前创建快照。

Related Skills

相关技能

  • rag-infrastructure - Full RAG pipeline
  • databases - General database management
  • postgresql - pgvector host database ops
  • rag-infrastructure - 完整RAG流水线
  • databases - 通用数据库管理
  • postgresql - pgvector宿主数据库运维