databricks-dbsql
Compare original and translation side by side
🇺🇸
Original
English🇨🇳
Translation
ChineseDatabricks SQL (DBSQL) - Advanced Features
Databricks SQL (DBSQL) - 高级功能
Quick Reference
快速参考
| Feature | Key Syntax | Since | Reference |
|---|---|---|---|
| SQL Scripting | | DBR 16.3+ | references/sql-scripting.md |
| Stored Procedures | | DBR 17.0+ | references/sql-scripting.md |
| Recursive CTEs | | DBR 17.0+ | references/sql-scripting.md |
| Transactions | | Preview | references/sql-scripting.md |
| Materialized Views | | Pro/Serverless | references/materialized-views-pipes.md |
| Temp Tables | | All | references/materialized-views-pipes.md |
| Pipe Syntax | | DBR 16.1+ | references/materialized-views-pipes.md |
| Geospatial (H3) | | DBR 11.2+ | references/geospatial-collations.md |
| Geospatial (ST) | | DBR 16.0+ | references/geospatial-collations.md |
| Collations | | DBR 16.1+ | references/geospatial-collations.md |
| AI Functions | | DBR 15.1+ | references/ai-functions.md |
| http_request | | Pro/Serverless | references/ai-functions.md |
| remote_query | | Pro/Serverless | references/ai-functions.md |
| read_files | | All | references/ai-functions.md |
| Data Modeling | Star schema, Liquid Clustering | All | references/best-practices.md |
| 特性 | 核心语法 | DBR版本 | 参考链接 |
|---|---|---|---|
| SQL 脚本 | | DBR 16.3+ | references/sql-scripting.md |
| 存储过程 | | DBR 17.0+ | references/sql-scripting.md |
| 递归CTE | | DBR 17.0+ | references/sql-scripting.md |
| 事务 | | 预览版 | references/sql-scripting.md |
| 物化视图 | | Pro/Serverless | references/materialized-views-pipes.md |
| 临时表 | | 所有版本 | references/materialized-views-pipes.md |
| 管道语法 | | DBR 16.1+ | references/materialized-views-pipes.md |
| 空间数据(H3) | | DBR 11.2+ | references/geospatial-collations.md |
| 空间数据(ST) | | DBR 16.0+ | references/geospatial-collations.md |
| 排序规则 | | DBR 16.1+ | references/geospatial-collations.md |
| AI函数 | | DBR 15.1+ | references/ai-functions.md |
| http_request | | Pro/Serverless | references/ai-functions.md |
| remote_query | | Pro/Serverless | references/ai-functions.md |
| read_files | | 所有版本 | references/ai-functions.md |
| 数据建模 | 星型模型、Liquid Clustering | 所有版本 | references/best-practices.md |
Common Patterns
常见模式
SQL Scripting - Procedural ETL
SQL脚本 - 过程式ETL
sql
BEGIN
DECLARE v_count INT;
DECLARE v_status STRING DEFAULT 'pending';
SET v_count = (SELECT COUNT(*) FROM catalog.schema.raw_orders WHERE status = 'new');
IF v_count > 0 THEN
INSERT INTO catalog.schema.processed_orders
SELECT *, current_timestamp() AS processed_at
FROM catalog.schema.raw_orders
WHERE status = 'new';
SET v_status = 'completed';
ELSE
SET v_status = 'skipped';
END IF;
SELECT v_status AS result, v_count AS rows_processed;
ENDsql
BEGIN
DECLARE v_count INT;
DECLARE v_status STRING DEFAULT 'pending';
SET v_count = (SELECT COUNT(*) FROM catalog.schema.raw_orders WHERE status = 'new');
IF v_count > 0 THEN
INSERT INTO catalog.schema.processed_orders
SELECT *, current_timestamp() AS processed_at
FROM catalog.schema.raw_orders
WHERE status = 'new';
SET v_status = 'completed';
ELSE
SET v_status = 'skipped';
END IF;
SELECT v_status AS result, v_count AS rows_processed;
ENDStored Procedure with Error Handling
带错误处理的存储过程
sql
CREATE OR REPLACE PROCEDURE catalog.schema.upsert_customers(
IN p_source STRING,
OUT p_rows_affected INT
)
LANGUAGE SQL
SQL SECURITY INVOKER
BEGIN
DECLARE EXIT HANDLER FOR SQLEXCEPTION
BEGIN
SET p_rows_affected = -1;
SIGNAL SQLSTATE '45000'
SET MESSAGE_TEXT = concat('Upsert failed for source: ', p_source);
END;
MERGE INTO catalog.schema.dim_customer AS t
USING (SELECT * FROM identifier(p_source)) AS s
ON t.customer_id = s.customer_id
WHEN MATCHED THEN UPDATE SET *
WHEN NOT MATCHED THEN INSERT *;
SET p_rows_affected = (SELECT COUNT(*) FROM identifier(p_source));
END;
-- Invoke:
CALL catalog.schema.upsert_customers('catalog.schema.staging_customers', ?);sql
CREATE OR REPLACE PROCEDURE catalog.schema.upsert_customers(
IN p_source STRING,
OUT p_rows_affected INT
)
LANGUAGE SQL
SQL SECURITY INVOKER
BEGIN
DECLARE EXIT HANDLER FOR SQLEXCEPTION
BEGIN
SET p_rows_affected = -1;
SIGNAL SQLSTATE '45000'
SET MESSAGE_TEXT = concat('Upsert failed for source: ', p_source);
END;
MERGE INTO catalog.schema.dim_customer AS t
USING (SELECT * FROM identifier(p_source)) AS s
ON t.customer_id = s.customer_id
WHEN MATCHED THEN UPDATE SET *
WHEN NOT MATCHED THEN INSERT *;
SET p_rows_affected = (SELECT COUNT(*) FROM identifier(p_source));
END;
-- 调用:
CALL catalog.schema.upsert_customers('catalog.schema.staging_customers', ?);Materialized View with Scheduled Refresh
带定时刷新的物化视图
sql
CREATE OR REPLACE MATERIALIZED VIEW catalog.schema.daily_revenue
CLUSTER BY (order_date)
SCHEDULE EVERY 1 HOUR
COMMENT 'Hourly-refreshed daily revenue by region'
AS SELECT
order_date,
region,
SUM(amount) AS total_revenue,
COUNT(DISTINCT customer_id) AS unique_customers
FROM catalog.schema.fact_orders
JOIN catalog.schema.dim_store USING (store_id)
GROUP BY order_date, region;sql
CREATE OR REPLACE MATERIALIZED VIEW catalog.schema.daily_revenue
CLUSTER BY (order_date)
SCHEDULE EVERY 1 HOUR
COMMENT 'Hourly-refreshed daily revenue by region'
AS SELECT
order_date,
region,
SUM(amount) AS total_revenue,
COUNT(DISTINCT customer_id) AS unique_customers
FROM catalog.schema.fact_orders
JOIN catalog.schema.dim_store USING (store_id)
GROUP BY order_date, region;Pipe Syntax - Readable Transformations
管道语法 - 可读性更强的转换
sql
-- Traditional SQL rewritten with pipe syntax
FROM catalog.schema.fact_orders
|> WHERE order_date >= current_date() - INTERVAL 30 DAYS
|> AGGREGATE SUM(amount) AS total, COUNT(*) AS cnt GROUP BY region, product_category
|> WHERE total > 10000
|> ORDER BY total DESC
|> LIMIT 20;sql
-- 用管道语法重写传统SQL
FROM catalog.schema.fact_orders
|> WHERE order_date >= current_date() - INTERVAL 30 DAYS
|> AGGREGATE SUM(amount) AS total, COUNT(*) AS cnt GROUP BY region, product_category
|> WHERE total > 10000
|> ORDER BY total DESC
|> LIMIT 20;AI Functions - Enrich Data with LLMs
AI函数 - 用大语言模型丰富数据
sql
-- Classify support tickets
SELECT
ticket_id,
description,
ai_classify(description, ARRAY('billing', 'technical', 'account', 'feature_request')) AS category,
ai_analyze_sentiment(description) AS sentiment
FROM catalog.schema.support_tickets
LIMIT 100;
-- Extract entities from text
SELECT
doc_id,
ai_extract(content, ARRAY('person_name', 'company', 'dollar_amount')) AS entities
FROM catalog.schema.contracts;
-- General-purpose AI query with structured output
SELECT ai_query(
'databricks-meta-llama-3-3-70b-instruct',
concat('Summarize this customer feedback in JSON with keys: topic, sentiment, action_items. Feedback: ', feedback),
returnType => 'STRUCT<topic STRING, sentiment STRING, action_items ARRAY<STRING>>'
) AS analysis
FROM catalog.schema.customer_feedback
LIMIT 50;sql
-- 分类支持工单
SELECT
ticket_id,
description,
ai_classify(description, ARRAY('billing', 'technical', 'account', 'feature_request')) AS category,
ai_analyze_sentiment(description) AS sentiment
FROM catalog.schema.support_tickets
LIMIT 100;
-- 从文本中提取实体
SELECT
doc_id,
ai_extract(content, ARRAY('person_name', 'company', 'dollar_amount')) AS entities
FROM catalog.schema.contracts;
-- 带结构化输出的通用AI查询
SELECT ai_query(
'databricks-meta-llama-3-3-70b-instruct',
concat('Summarize this customer feedback in JSON with keys: topic, sentiment, action_items. Feedback: ', feedback),
returnType => 'STRUCT<topic STRING, sentiment STRING, action_items ARRAY<STRING>>'
) AS analysis
FROM catalog.schema.customer_feedback
LIMIT 50;Geospatial - Proximity Search with H3
空间数据 - 基于H3的邻近搜索
sql
-- Find stores within 5km of each customer using H3 indexing
WITH customer_h3 AS (
SELECT *, h3_longlatash3(longitude, latitude, 7) AS h3_cell
FROM catalog.schema.customers
),
store_h3 AS (
SELECT *, h3_longlatash3(longitude, latitude, 7) AS h3_cell
FROM catalog.schema.stores
)
SELECT
c.customer_id,
s.store_id,
ST_Distance(
ST_Point(c.longitude, c.latitude),
ST_Point(s.longitude, s.latitude)
) AS distance_m
FROM customer_h3 c
JOIN store_h3 s ON h3_ischildof(c.h3_cell, h3_toparent(s.h3_cell, 5))
WHERE ST_Distance(
ST_Point(c.longitude, c.latitude),
ST_Point(s.longitude, s.latitude)
) < 5000;sql
-- 使用H3索引查找每个客户5公里范围内的门店
WITH customer_h3 AS (
SELECT *, h3_longlatash3(longitude, latitude, 7) AS h3_cell
FROM catalog.schema.customers
),
store_h3 AS (
SELECT *, h3_longlatash3(longitude, latitude, 7) AS h3_cell
FROM catalog.schema.stores
)
SELECT
c.customer_id,
s.store_id,
ST_Distance(
ST_Point(c.longitude, c.latitude),
ST_Point(s.longitude, s.latitude)
) AS distance_m
FROM customer_h3 c
JOIN store_h3 s ON h3_ischildof(c.h3_cell, h3_toparent(s.h3_cell, 5))
WHERE ST_Distance(
ST_Point(c.longitude, c.latitude),
ST_Point(s.longitude, s.latitude)
) < 5000;Collation - Case-Insensitive Search
排序规则 - 大小写不敏感搜索
sql
-- Create table with case-insensitive collation
CREATE TABLE catalog.schema.products (
product_id BIGINT GENERATED ALWAYS AS IDENTITY,
name STRING COLLATE UTF8_LCASE,
category STRING COLLATE UTF8_LCASE,
price DECIMAL(10, 2)
);
-- Queries automatically case-insensitive (no LOWER() needed)
SELECT * FROM catalog.schema.products
WHERE name = 'MacBook Pro'; -- matches 'macbook pro', 'MACBOOK PRO', etc.sql
-- 创建带大小写不敏感排序规则的表
CREATE TABLE catalog.schema.products (
product_id BIGINT GENERATED ALWAYS AS IDENTITY,
name STRING COLLATE UTF8_LCASE,
category STRING COLLATE UTF8_LCASE,
price DECIMAL(10, 2)
);
-- 查询自动支持大小写不敏感(无需LOWER())
SELECT * FROM catalog.schema.products
WHERE name = 'MacBook Pro'; -- 匹配'macbook pro'、'MACBOOK PRO'等http_request - Call External APIs
http_request - 调用外部API
sql
-- Set up connection first (one-time)
CREATE CONNECTION my_api_conn
TYPE HTTP
OPTIONS (host 'https://api.example.com', bearer_token secret('scope', 'token'));
-- Call API from SQL
SELECT
order_id,
http_request(
conn => 'my_api_conn',
method => 'POST',
path => '/v1/validate',
json => to_json(named_struct('order_id', order_id, 'amount', amount))
).text AS api_response
FROM catalog.schema.orders
WHERE needs_validation = true;sql
-- 先设置连接(一次性操作)
CREATE CONNECTION my_api_conn
TYPE HTTP
OPTIONS (host 'https://api.example.com', bearer_token secret('scope', 'token'));
-- 从SQL中调用API
SELECT
order_id,
http_request(
conn => 'my_api_conn',
method => 'POST',
path => '/v1/validate',
json => to_json(named_struct('order_id', order_id, 'amount', amount))
).text AS api_response
FROM catalog.schema.orders
WHERE needs_validation = true;read_files - Ingest Raw Files
read_files - 导入原始文件
sql
-- Read JSON files from a Volume with schema hints
SELECT *
FROM read_files(
'/Volumes/catalog/schema/raw/events/',
format => 'json',
schemaHints => 'event_id STRING, timestamp TIMESTAMP, payload MAP<STRING, STRING>',
pathGlobFilter => '*.json',
recursiveFileLookup => true
);
-- Read CSV with options
SELECT *
FROM read_files(
'/Volumes/catalog/schema/raw/sales/',
format => 'csv',
header => true,
delimiter => '|',
dateFormat => 'yyyy-MM-dd',
schema => 'sale_id INT, sale_date DATE, amount DECIMAL(10,2), store STRING'
);sql
-- 从Volume读取JSON文件并指定 schema 提示
SELECT *
FROM read_files(
'/Volumes/catalog/schema/raw/events/',
format => 'json',
schemaHints => 'event_id STRING, timestamp TIMESTAMP, payload MAP<STRING, STRING>',
pathGlobFilter => '*.json',
recursiveFileLookup => true
);
-- 读取CSV文件并配置参数
SELECT *
FROM read_files(
'/Volumes/catalog/schema/raw/sales/',
format => 'csv',
header => true,
delimiter => '|',
dateFormat => 'yyyy-MM-dd',
schema => 'sale_id INT, sale_date DATE, amount DECIMAL(10,2), store STRING'
);Recursive CTE - Hierarchy Traversal
递归CTE - 层级遍历
sql
WITH RECURSIVE org_chart AS (
-- Anchor: top-level managers
SELECT employee_id, name, manager_id, 0 AS depth, ARRAY(name) AS path
FROM catalog.schema.employees
WHERE manager_id IS NULL
UNION ALL
-- Recursive: direct reports
SELECT e.employee_id, e.name, e.manager_id, o.depth + 1, array_append(o.path, e.name)
FROM catalog.schema.employees e
JOIN org_chart o ON e.manager_id = o.employee_id
WHERE o.depth < 10 -- safety limit
)
SELECT * FROM org_chart ORDER BY depth, name;sql
WITH RECURSIVE org_chart AS (
-- 锚点:顶层管理者
SELECT employee_id, name, manager_id, 0 AS depth, ARRAY(name) AS path
FROM catalog.schema.employees
WHERE manager_id IS NULL
UNION ALL
-- 递归:直接下属
SELECT e.employee_id, e.name, e.manager_id, o.depth + 1, array_append(o.path, e.name)
FROM catalog.schema.employees e
JOIN org_chart o ON e.manager_id = o.employee_id
WHERE o.depth < 10 -- 安全限制
)
SELECT * FROM org_chart ORDER BY depth, name;remote_query - Federated Queries
remote_query - 联邦查询
sql
-- Query PostgreSQL via Lakehouse Federation
SELECT *
FROM remote_query(
'my_postgres_connection',
database => 'my_database',
query => 'SELECT customer_id, email, created_at FROM customers WHERE active = true'
);sql
-- 通过Lakehouse Federation查询PostgreSQL
SELECT *
FROM remote_query(
'my_postgres_connection',
database => 'my_database',
query => 'SELECT customer_id, email, created_at FROM customers WHERE active = true'
);Reference Files
参考文档
Load these for detailed syntax, full parameter lists, and advanced patterns:
| File | Contents | When to Read |
|---|---|---|
| references/sql-scripting.md | SQL Scripting, Stored Procedures, Recursive CTEs, Transactions | User needs procedural SQL, error handling, loops, dynamic SQL |
| references/materialized-views-pipes.md | Materialized Views, Temp Tables/Views, Pipe Syntax | User needs MVs, refresh scheduling, temp objects, pipe operator |
| references/geospatial-collations.md | 39 H3 functions, 80+ ST functions, Collation types and hierarchy | User needs spatial analysis, H3 indexing, case/accent handling |
| references/ai-functions.md | 13 AI functions, http_request, remote_query, read_files (all options) | User needs AI enrichment, API calls, federation, file ingestion |
| references/best-practices.md | Data modeling, performance, Liquid Clustering, anti-patterns | User needs architecture guidance, optimization, or modeling advice |
加载以下文档获取详细语法、完整参数列表和高级模式:
| 文件 | 内容 | 阅读场景 |
|---|---|---|
| references/sql-scripting.md | SQL脚本、存储过程、递归CTE、事务 | 用户需要过程式SQL、错误处理、循环、动态SQL时 |
| references/materialized-views-pipes.md | 物化视图、临时表/视图、管道语法 | 用户需要物化视图、刷新调度、临时对象、管道运算符时 |
| references/geospatial-collations.md | 39个H3函数、80+个ST函数、排序规则类型与层级 | 用户需要空间分析、H3索引、大小写/重音处理时 |
| references/ai-functions.md | 13个AI函数、http_request、remote_query、read_files(所有参数) | 用户需要AI增强、API调用、联邦查询、文件导入时 |
| references/best-practices.md | 数据建模、性能优化、Liquid Clustering、反模式 | 用户需要架构指导、优化建议或建模方案时 |
Key Guidelines
核心指南
- Always use Serverless SQL warehouses for AI functions, MVs, and http_request
- Use during development with AI functions to control costs
LIMIT - Prefer Liquid Clustering over partitioning for new tables (1-4 keys max)
- Use when unsure about clustering keys
CLUSTER BY AUTO - Star schema in Gold layer for BI; OBT acceptable in Silver
- Define PK/FK constraints on dimensional models for query optimization
- Use for user-facing string columns that need case-insensitive search
COLLATE UTF8_LCASE - Test SQL via CLI () or notebooks before deploying. If
databricks experimental aitools tools queryis rejected on your CLI version, set--warehousein the environment instead.DATABRICKS_WAREHOUSE_ID
- AI函数、物化视图和http_request请始终使用Serverless SQL仓库
- **开发AI函数时使用**以控制成本
LIMIT - 新表优先使用Liquid Clustering而非分区(最多1-4个键)
- 不确定聚类键时使用
CLUSTER BY AUTO - Gold层使用星型模型用于BI分析;Silver层可使用OBT(宽表)
- 在维度模型上定义PK/FK约束以优化查询
- **用户可见的字符串列使用**以支持大小写不敏感搜索
COLLATE UTF8_LCASE - 部署前通过CLI()或Notebook测试SQL。如果你的CLI版本不支持
databricks experimental aitools tools query参数,请在环境变量中设置--warehouse。DATABRICKS_WAREHOUSE_ID