Back to Details

databricks-vector-search

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

Databricks Vector Search

Databricks Vector Search

FIRST: Use the parent 
databricks-core
skill for CLI basics, authentication, and profile selection.
Patterns for creating, managing, and querying vector search indexes for RAG and semantic search applications.
注意: 请先使用父级
databricks-core
技能了解CLI基础、认证和配置文件选择。
本文介绍了为RAG和语义搜索应用创建、管理和查询向量搜索索引的模式。

When to Use

适用场景

Use this skill when:
  • Building RAG (Retrieval-Augmented Generation) applications
  • Implementing semantic search or similarity matching
  • Creating vector indexes from Delta tables
  • Choosing between storage-optimized and standard endpoints
  • Querying vector indexes with filters
在以下场景中使用本技能：
  • 构建RAG（检索增强生成）应用
  • 实现语义搜索或相似度匹配
  • 从Delta表创建向量索引
  • 在存储优化型和标准型端点之间做选择
  • 使用过滤器查询向量索引

Overview

概述

Databricks Vector Search provides managed vector similarity search with automatic embedding generation and Delta Lake integration.
ComponentDescription
EndpointCompute resource hosting indexes (Standard or Storage-Optimized)
IndexVector data structure for similarity search
Delta SyncAuto-syncs with source Delta table
Direct AccessManual CRUD operations on vectors
Databricks Vector Search提供托管式向量相似度搜索功能，支持自动嵌入生成和Delta Lake集成。
组件说明
Endpoint（端点）托管索引的计算资源（标准型或存储优化型）
Index（索引）用于相似度搜索的向量数据结构
Delta Sync（Delta同步）与源Delta表自动同步
Direct Access（直接访问）对向量执行手动CRUD操作

Endpoint Types

端点类型

TypeLatencyCapacityCostBest For
Standard20-50ms320M vectors (768 dim)HigherReal-time, low-latency
Storage-Optimized300-500ms1B+ vectors (768 dim)7x lowerLarge-scale, cost-sensitive
类型延迟容量成本最佳适用场景
Standard（标准型）20-50ms3.2亿个向量（768维度）较高实时、低延迟场景
Storage-Optimized（存储优化型）300-500ms10亿+个向量（768维度）低7倍大规模、成本敏感型场景

Index Types

索引类型

TypeEmbeddingsSyncUse Case
Delta Sync (managed)Databricks computesAuto from DeltaEasiest setup
Delta Sync (self-managed)You provideAuto from DeltaCustom embeddings
Direct AccessYou provideManual CRUDReal-time updates
类型嵌入向量同步方式适用场景
Delta Sync（托管式）由Databricks计算生成从Delta表自动同步最简便的设置方式
Delta Sync（自托管式）由用户提供从Delta表自动同步自定义嵌入向量场景
Direct Access（直接访问）由用户提供手动CRUD操作实时更新场景

Quick Start

快速开始

Create Endpoint

创建端点

python
from databricks.sdk import WorkspaceClient

w = WorkspaceClient()
python
from databricks.sdk import WorkspaceClient

w = WorkspaceClient()

Create a standard endpoint

创建标准型端点

endpoint = w.vector_search_endpoints.create_endpoint( name="my-vs-endpoint", endpoint_type="STANDARD" # or "STORAGE_OPTIMIZED" )
endpoint = w.vector_search_endpoints.create_endpoint( name="my-vs-endpoint", endpoint_type="STANDARD" # 或 "STORAGE_OPTIMIZED" )

Note: Endpoint creation is asynchronous; check status with get_endpoint()

注意：端点创建为异步操作；可使用get_endpoint()检查状态

undefined
undefined

Create Delta Sync Index (Managed Embeddings)

创建Delta Sync索引（托管式嵌入向量）

python
undefined
python
undefined

Source table must have: primary key column + text column

源表必须包含：主键列 + 文本列

index = w.vector_search_indexes.create_index( name="catalog.schema.my_index", endpoint_name="my-vs-endpoint", primary_key="id", index_type="DELTA_SYNC", delta_sync_index_spec={ "source_table": "catalog.schema.documents", "embedding_source_columns": [ { "name": "content", # Text column to embed "embedding_model_endpoint_name": "databricks-gte-large-en" } ], "pipeline_type": "TRIGGERED" # or "CONTINUOUS" } )
undefined
index = w.vector_search_indexes.create_index( name="catalog.schema.my_index", endpoint_name="my-vs-endpoint", primary_key="id", index_type="DELTA_SYNC", delta_sync_index_spec={ "source_table": "catalog.schema.documents", "embedding_source_columns": [ { "name": "content", # 要生成嵌入向量的文本列 "embedding_model_endpoint_name": "databricks-gte-large-en" } ], "pipeline_type": "TRIGGERED" # 或 "CONTINUOUS" } )
undefined

Query Index

查询索引

python
results = w.vector_search_indexes.query_index(
    index_name="catalog.schema.my_index",
    columns=["id", "content", "metadata"],
    query_text="What is machine learning?",
    num_results=5
)

for doc in results.result.data_array:
    score = doc[-1]  # Similarity score is last column
    print(f"Score: {score}, Content: {doc[1][:100]}...")
python
results = w.vector_search_indexes.query_index(
    index_name="catalog.schema.my_index",
    columns=["id", "content", "metadata"],
    query_text="What is machine learning?",
    num_results=5
)

for doc in results.result.data_array:
    score = doc[-1]  # 相似度分数为最后一列
    print(f"Score: {score}, Content: {doc[1][:100]}...")

Common Patterns

常见模式

Create Storage-Optimized Endpoint

创建存储优化型端点

python
undefined
python
undefined

For large-scale, cost-effective deployments

适用于大规模、高性价比部署

endpoint = w.vector_search_endpoints.create_endpoint( name="my-storage-endpoint", endpoint_type="STORAGE_OPTIMIZED" )
undefined
endpoint = w.vector_search_endpoints.create_endpoint( name="my-storage-endpoint", endpoint_type="STORAGE_OPTIMIZED" )
undefined

Delta Sync with Self-Managed Embeddings

自托管嵌入向量的Delta Sync

python
undefined
python
undefined

Source table must have: primary key + embedding vector column

源表必须包含：主键 + 嵌入向量列

index = w.vector_search_indexes.create_index( name="catalog.schema.my_index", endpoint_name="my-vs-endpoint", primary_key="id", index_type="DELTA_SYNC", delta_sync_index_spec={ "source_table": "catalog.schema.documents", "embedding_vector_columns": [ { "name": "embedding", # Pre-computed embedding column "embedding_dimension": 768 } ], "pipeline_type": "TRIGGERED" } )
undefined
index = w.vector_search_indexes.create_index( name="catalog.schema.my_index", endpoint_name="my-vs-endpoint", primary_key="id", index_type="DELTA_SYNC", delta_sync_index_spec={ "source_table": "catalog.schema.documents", "embedding_vector_columns": [ { "name": "embedding", # 预计算的嵌入向量列 "embedding_dimension": 768 } ], "pipeline_type": "TRIGGERED" } )
undefined

Direct Access Index

直接访问索引

python
import json
python
import json

Create index for manual CRUD

创建用于手动CRUD的索引

index = w.vector_search_indexes.create_index( name="catalog.schema.direct_index", endpoint_name="my-vs-endpoint", primary_key="id", index_type="DIRECT_ACCESS", direct_access_index_spec={ "embedding_vector_columns": [ {"name": "embedding", "embedding_dimension": 768} ], "schema_json": json.dumps({ "id": "string", "text": "string", "embedding": "array<float>", "metadata": "string" }) } )
index = w.vector_search_indexes.create_index( name="catalog.schema.direct_index", endpoint_name="my-vs-endpoint", primary_key="id", index_type="DIRECT_ACCESS", direct_access_index_spec={ "embedding_vector_columns": [ {"name": "embedding", "embedding_dimension": 768} ], "schema_json": json.dumps({ "id": "string", "text": "string", "embedding": "array<float>", "metadata": "string" }) } )

Upsert data

插入/更新数据

w.vector_search_indexes.upsert_data_vector_index( index_name="catalog.schema.direct_index", inputs_json=json.dumps([ {"id": "1", "text": "Hello", "embedding": [0.1, 0.2, ...], "metadata": "doc1"}, {"id": "2", "text": "World", "embedding": [0.3, 0.4, ...], "metadata": "doc2"}, ]) )
w.vector_search_indexes.upsert_data_vector_index( index_name="catalog.schema.direct_index", inputs_json=json.dumps([ {"id": "1", "text": "Hello", "embedding": [0.1, 0.2, ...], "metadata": "doc1"}, {"id": "2", "text": "World", "embedding": [0.3, 0.4, ...], "metadata": "doc2"}, ]) )

Delete data

删除数据

w.vector_search_indexes.delete_data_vector_index( index_name="catalog.schema.direct_index", primary_keys=["1", "2"] )
undefined
w.vector_search_indexes.delete_data_vector_index( index_name="catalog.schema.direct_index", primary_keys=["1", "2"] )
undefined

Query with Embedding Vector

使用嵌入向量查询

python
undefined
python
undefined

When you have pre-computed query embedding

当你有预计算的查询嵌入向量时

results = w.vector_search_indexes.query_index( index_name="catalog.schema.my_index", columns=["id", "text"], query_vector=[0.1, 0.2, 0.3, ...], # Your 768-dim vector num_results=10 )
undefined
results = w.vector_search_indexes.query_index( index_name="catalog.schema.my_index", columns=["id", "text"], query_vector=[0.1, 0.2, 0.3, ...], # 你的768维度向量 num_results=10 )
undefined

Hybrid Search (Semantic + Keyword)

混合搜索（语义+关键词）

Hybrid search combines vector similarity (ANN) with BM25 keyword scoring. Use it when queries contain exact terms that must match — SKUs, error codes, proper nouns, or technical terminology — where pure semantic search might miss keyword-specific results. See references/search-modes.md for detailed guidance on choosing between ANN and hybrid search.
python
undefined
混合搜索将向量相似度（ANN）与BM25关键词评分相结合。当查询包含必须匹配的精确术语（如SKU、错误代码、专有名词或技术术语）时使用此模式，因为纯语义搜索可能会遗漏关键词相关结果。如需了解在ANN和混合搜索之间做选择的详细指南，请参阅references/search-modes.md。
python
undefined

Combines vector similarity with keyword matching

结合向量相似度与关键词匹配

results = w.vector_search_indexes.query_index( index_name="catalog.schema.my_index", columns=["id", "content"], query_text="SPARK-12345 executor memory error", query_type="HYBRID", num_results=10 )
undefined
results = w.vector_search_indexes.query_index( index_name="catalog.schema.my_index", columns=["id", "content"], query_text="SPARK-12345 executor memory error", query_type="HYBRID", num_results=10 )
undefined

Filtering

过滤功能

Standard Endpoint Filters (Dictionary)

标准型端点过滤器（字典格式）

python
undefined
python
undefined

filters_json uses dictionary format

filters_json使用字典格式

results = w.vector_search_indexes.query_index( index_name="catalog.schema.my_index", columns=["id", "content"], query_text="machine learning", num_results=10, filters_json='{"category": "ai", "status": ["active", "pending"]}' )
undefined
results = w.vector_search_indexes.query_index( index_name="catalog.schema.my_index", columns=["id", "content"], query_text="machine learning", num_results=10, filters_json='{"category": "ai", "status": ["active", "pending"]}' )
undefined

Storage-Optimized Filters (SQL-like)

存储优化型端点过滤器（类SQL格式）

Storage-Optimized endpoints use SQL-like filter syntax via the 
databricks-vectorsearch
package's 
filters
parameter (accepts a string):
python
from databricks.vector_search.client import VectorSearchClient

vsc = VectorSearchClient()
index = vsc.get_index(endpoint_name="my-storage-endpoint", index_name="catalog.schema.my_index")
存储优化型端点通过
databricks-vectorsearch
包的
filters
参数使用类SQL过滤语法（接受字符串）：
python
from databricks.vector_search.client import VectorSearchClient

vsc = VectorSearchClient()
index = vsc.get_index(endpoint_name="my-storage-endpoint", index_name="catalog.schema.my_index")

SQL-like filter syntax for storage-optimized endpoints

存储优化型端点使用类SQL过滤语法

results = index.similarity_search( query_text="machine learning", columns=["id", "content"], num_results=10, filters="category = 'ai' AND status IN ('active', 'pending')" )
results = index.similarity_search( query_text="machine learning", columns=["id", "content"], num_results=10, filters="category = 'ai' AND status IN ('active', 'pending')" )

More filter examples

更多过滤器示例

filters="price > 100 AND price < 500"

filters="price > 100 AND price < 500"

filters="department LIKE 'eng%'"

filters="department LIKE 'eng%'"

filters="created_at >= '2024-01-01'"

filters="created_at >= '2024-01-01'"

undefined
undefined

Trigger Index Sync

触发索引同步

python
undefined
python
undefined

For TRIGGERED pipeline type, manually sync

对于TRIGGERED管道类型，手动触发同步

w.vector_search_indexes.sync_index( index_name="catalog.schema.my_index" )
undefined
w.vector_search_indexes.sync_index( index_name="catalog.schema.my_index" )
undefined

Scan All Index Entries

扫描所有索引条目

python
undefined
python
undefined

Retrieve all vectors (for debugging/export)

获取所有向量（用于调试/导出）

scan_result = w.vector_search_indexes.scan_index( index_name="catalog.schema.my_index", num_results=100 )
undefined
scan_result = w.vector_search_indexes.scan_index( index_name="catalog.schema.my_index", num_results=100 )
undefined

Reference Files

参考文件

TopicFileDescription
Index Typesreferences/index-types.mdDetailed comparison of Delta Sync (managed/self-managed) vs Direct Access
End-to-End RAGreferences/end-to-end-rag.mdComplete walkthrough: source table → endpoint → index → query → agent integration
Search Modesreferences/search-modes.mdWhen to use semantic (ANN) vs hybrid search, decision guide
Operationsreferences/troubleshooting-and-operations.mdMonitoring, cost optimization, capacity planning, migration
主题文件说明
索引类型references/index-types.mdDelta Sync（托管/自托管）与Direct Access的详细对比
端到端RAGreferences/end-to-end-rag.md完整流程指南：源表 → 端点 → 索引 → 查询 → Agent集成
搜索模式references/search-modes.md何时使用语义（ANN）搜索vs混合搜索的决策指南
运维操作references/troubleshooting-and-operations.md监控、成本优化、容量规划、迁移

CLI Quick Reference

CLI快速参考

bash
undefined
bash
undefined

List endpoints

列出端点

databricks vector-search-endpoints list-endpoints
databricks vector-search-endpoints list-endpoints

Create endpoint (positional args: NAME ENDPOINT_TYPE)

创建端点（位置参数：名称 端点类型）

databricks vector-search-endpoints create-endpoint my-endpoint STANDARD
databricks vector-search-endpoints create-endpoint my-endpoint STANDARD

List indexes on endpoint (positional arg: ENDPOINT_NAME)

列出端点上的索引（位置参数：端点名称）

databricks vector-search-indexes list-indexes my-endpoint
databricks vector-search-indexes list-indexes my-endpoint

Get index status (positional arg: INDEX_NAME)

获取索引状态（位置参数：索引名称）

databricks vector-search-indexes get-index catalog.schema.my_index
databricks vector-search-indexes get-index catalog.schema.my_index

Sync index (positional arg: INDEX_NAME)

同步索引（位置参数：索引名称）

databricks vector-search-indexes sync-index catalog.schema.my_index
databricks vector-search-indexes sync-index catalog.schema.my_index

Delete index (positional arg: INDEX_NAME)

删除索引（位置参数：索引名称）

databricks vector-search-indexes delete-index catalog.schema.my_index
undefined
databricks vector-search-indexes delete-index catalog.schema.my_index
undefined

Common Issues

常见问题

IssueSolution
Index sync slowUse Storage-Optimized endpoints (20x faster indexing)
Query latency highUse Standard endpoint for <100ms latency
filters_json not workingStorage-Optimized uses SQL-like string filters via 
databricks-vectorsearch
package's 
filters
parameter
Embedding dimension mismatchEnsure query and index dimensions match
Index not updatingCheck pipeline_type; use sync_index() for TRIGGERED
Out of capacityUpgrade to Storage-Optimized (1B+ vectors)
query_vector
truncated		Large vectors (e.g. 1024-dim) can be truncated when serialized as JSON. Use 
query_text
instead (for managed embedding indexes), or use the Databricks SDK to pass raw vectors
问题解决方案
索引同步缓慢使用存储优化型端点（索引速度快20倍）
查询延迟高使用标准型端点实现<100ms延迟
filters_json不生效存储优化型端点需通过
databricks-vectorsearch
包的
filters
参数使用类SQL字符串过滤器
嵌入向量维度不匹配确保查询向量与索引向量维度一致
索引未更新检查pipeline_type；对于TRIGGERED类型，使用sync_index()手动同步
容量不足升级到存储优化型端点（支持10亿+向量）
query_vector
被截断		大向量（如1024维度）序列化为JSON时可能被截断。对于托管嵌入索引，请改用
query_text
；或使用Databricks SDK传递原始向量

Embedding Models

嵌入模型

Databricks provides built-in embedding models:
ModelDimensionsContext WindowUse Case
databricks-gte-large-en
10248192 tokensEnglish text, high quality
databricks-bge-large-en
1024512 tokensEnglish text, general purpose
python
undefined
Databricks提供内置嵌入模型：
模型维度上下文窗口适用场景
databricks-gte-large-en
10248192 tokens英文文本，高质量
databricks-bge-large-en
1024512 tokens英文文本，通用场景
python
undefined

Use with managed embeddings

与托管嵌入向量配合使用

embedding_source_columns=[ { "name": "content", "embedding_model_endpoint_name": "databricks-gte-large-en" } ]
undefined
embedding_source_columns=[ { "name": "content", "embedding_model_endpoint_name": "databricks-gte-large-en" } ]
undefined

Notes

注意事项

  • Storage-Optimized is newer — better for most use cases unless you need <100ms latency
  • Delta Sync recommended — easier than Direct Access for most scenarios
  • Hybrid search — available for both Delta Sync and Direct Access indexes
  • columns_to_sync
    matters     — only synced columns are available in query results; include all columns you need
  • Filter syntax differs by endpoint — Standard uses dict-format filters, Storage-Optimized uses SQL-like string filters. Use the 
    databricks-vectorsearch
    package's 
    filters
    parameter which accepts both formats
  • Management vs runtime — CLI and SDK handle lifecycle management; for agent tool-calling at runtime, use 
    VectorSearchRetrieverTool
  • 存储优化型是较新的类型 —— 除需要<100ms延迟的场景外，大多数场景都更适用
  • 推荐使用Delta Sync —— 对于大多数场景，比Direct Access更简便
  • 混合搜索 —— Delta Sync和Direct Access索引均支持
  • columns_to_sync
    很重要     —— 只有同步的列可在查询结果中获取；请包含所有需要的列
  • 过滤器语法因端点类型而异 —— 标准型使用字典格式过滤器，存储优化型使用类SQL字符串过滤器。可使用
    databricks-vectorsearch
    包的
    filters
    参数，它同时支持两种格式
  • 管理与运行时 —— CLI和SDK负责生命周期管理；运行时Agent工具调用请使用
    VectorSearchRetrieverTool

Related Skills

相关技能

  • databricks-model-serving - Deploy agents that use VectorSearchRetrieverTool
  • databricks-agent-bricks - Knowledge Assistants use RAG over indexed documents
  • databricks-unstructured-pdf-generation - Generate documents to index in Vector Search
  • databricks-unity-catalog - Manage the catalogs and tables that back Delta Sync indexes
  • databricks-pipelines - Build Delta tables used as Vector Search sources
  • databricks-model-serving - 部署使用VectorSearchRetrieverTool的Agent
  • databricks-agent-bricks - 知识助手通过索引文档实现RAG
  • databricks-unstructured-pdf-generation - 生成可在Vector Search中索引的文档
  • databricks-unity-catalog - 管理支持Delta Sync索引的目录和表
  • databricks-pipelines - 构建用作Vector Search源的Delta表