databricks-dbsql

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

Databricks SQL (DBSQL) - Advanced Features

Databricks SQL (DBSQL) - 高级功能

Quick Reference

快速参考

FeatureKey SyntaxSinceReference
SQL Scripting
BEGIN...END
,
DECLARE
,
IF/WHILE/FOR
DBR 16.3+references/sql-scripting.md
Stored Procedures
CREATE PROCEDURE
,
CALL
DBR 17.0+references/sql-scripting.md
Recursive CTEs
WITH RECURSIVE
DBR 17.0+references/sql-scripting.md
Transactions
BEGIN ATOMIC...END
Previewreferences/sql-scripting.md
Materialized Views
CREATE MATERIALIZED VIEW
Pro/Serverlessreferences/materialized-views-pipes.md
Temp Tables
CREATE TEMPORARY TABLE
Allreferences/materialized-views-pipes.md
Pipe Syntax
|>
operator
DBR 16.1+references/materialized-views-pipes.md
Geospatial (H3)
h3_longlatash3()
,
h3_polyfillash3()
DBR 11.2+references/geospatial-collations.md
Geospatial (ST)
ST_Point()
,
ST_Contains()
, 80+ funcs
DBR 16.0+references/geospatial-collations.md
Collations
COLLATE
,
UTF8_LCASE
, locale-aware
DBR 16.1+references/geospatial-collations.md
AI Functions
ai_query()
,
ai_classify()
, 11+ funcs
DBR 15.1+references/ai-functions.md
http_request
http_request(conn, ...)
Pro/Serverlessreferences/ai-functions.md
remote_query
SELECT * FROM remote_query(...)
Pro/Serverlessreferences/ai-functions.md
read_files
SELECT * FROM read_files(...)
Allreferences/ai-functions.md
Data ModelingStar schema, Liquid ClusteringAllreferences/best-practices.md

特性核心语法DBR版本参考链接
SQL 脚本
BEGIN...END
,
DECLARE
,
IF/WHILE/FOR
DBR 16.3+references/sql-scripting.md
存储过程
CREATE PROCEDURE
,
CALL
DBR 17.0+references/sql-scripting.md
递归CTE
WITH RECURSIVE
DBR 17.0+references/sql-scripting.md
事务
BEGIN ATOMIC...END
预览版references/sql-scripting.md
物化视图
CREATE MATERIALIZED VIEW
Pro/Serverlessreferences/materialized-views-pipes.md
临时表
CREATE TEMPORARY TABLE
所有版本references/materialized-views-pipes.md
管道语法
|>
运算符
DBR 16.1+references/materialized-views-pipes.md
空间数据(H3)
h3_longlatash3()
,
h3_polyfillash3()
DBR 11.2+references/geospatial-collations.md
空间数据(ST)
ST_Point()
,
ST_Contains()
等80+函数
DBR 16.0+references/geospatial-collations.md
排序规则
COLLATE
,
UTF8_LCASE
, 区域感知
DBR 16.1+references/geospatial-collations.md
AI函数
ai_query()
,
ai_classify()
等11+函数
DBR 15.1+references/ai-functions.md
http_request
http_request(conn, ...)
Pro/Serverlessreferences/ai-functions.md
remote_query
SELECT * FROM remote_query(...)
Pro/Serverlessreferences/ai-functions.md
read_files
SELECT * FROM read_files(...)
所有版本references/ai-functions.md
数据建模星型模型、Liquid Clustering所有版本references/best-practices.md

Common Patterns

常见模式

SQL Scripting - Procedural ETL

SQL脚本 - 过程式ETL

sql
BEGIN
  DECLARE v_count INT;
  DECLARE v_status STRING DEFAULT 'pending';

  SET v_count = (SELECT COUNT(*) FROM catalog.schema.raw_orders WHERE status = 'new');

  IF v_count > 0 THEN
    INSERT INTO catalog.schema.processed_orders
    SELECT *, current_timestamp() AS processed_at
    FROM catalog.schema.raw_orders
    WHERE status = 'new';

    SET v_status = 'completed';
  ELSE
    SET v_status = 'skipped';
  END IF;

  SELECT v_status AS result, v_count AS rows_processed;
END
sql
BEGIN
  DECLARE v_count INT;
  DECLARE v_status STRING DEFAULT 'pending';

  SET v_count = (SELECT COUNT(*) FROM catalog.schema.raw_orders WHERE status = 'new');

  IF v_count > 0 THEN
    INSERT INTO catalog.schema.processed_orders
    SELECT *, current_timestamp() AS processed_at
    FROM catalog.schema.raw_orders
    WHERE status = 'new';

    SET v_status = 'completed';
  ELSE
    SET v_status = 'skipped';
  END IF;

  SELECT v_status AS result, v_count AS rows_processed;
END

Stored Procedure with Error Handling

带错误处理的存储过程

sql
CREATE OR REPLACE PROCEDURE catalog.schema.upsert_customers(
  IN p_source STRING,
  OUT p_rows_affected INT
)
LANGUAGE SQL
SQL SECURITY INVOKER
BEGIN
  DECLARE EXIT HANDLER FOR SQLEXCEPTION
  BEGIN
    SET p_rows_affected = -1;
    SIGNAL SQLSTATE '45000'
      SET MESSAGE_TEXT = concat('Upsert failed for source: ', p_source);
  END;

  MERGE INTO catalog.schema.dim_customer AS t
  USING (SELECT * FROM identifier(p_source)) AS s
  ON t.customer_id = s.customer_id
  WHEN MATCHED THEN UPDATE SET *
  WHEN NOT MATCHED THEN INSERT *;

  SET p_rows_affected = (SELECT COUNT(*) FROM identifier(p_source));
END;

-- Invoke:
CALL catalog.schema.upsert_customers('catalog.schema.staging_customers', ?);
sql
CREATE OR REPLACE PROCEDURE catalog.schema.upsert_customers(
  IN p_source STRING,
  OUT p_rows_affected INT
)
LANGUAGE SQL
SQL SECURITY INVOKER
BEGIN
  DECLARE EXIT HANDLER FOR SQLEXCEPTION
  BEGIN
    SET p_rows_affected = -1;
    SIGNAL SQLSTATE '45000'
      SET MESSAGE_TEXT = concat('Upsert failed for source: ', p_source);
  END;

  MERGE INTO catalog.schema.dim_customer AS t
  USING (SELECT * FROM identifier(p_source)) AS s
  ON t.customer_id = s.customer_id
  WHEN MATCHED THEN UPDATE SET *
  WHEN NOT MATCHED THEN INSERT *;

  SET p_rows_affected = (SELECT COUNT(*) FROM identifier(p_source));
END;

-- 调用:
CALL catalog.schema.upsert_customers('catalog.schema.staging_customers', ?);

Materialized View with Scheduled Refresh

带定时刷新的物化视图

sql
CREATE OR REPLACE MATERIALIZED VIEW catalog.schema.daily_revenue
  CLUSTER BY (order_date)
  SCHEDULE EVERY 1 HOUR
  COMMENT 'Hourly-refreshed daily revenue by region'
AS SELECT
    order_date,
    region,
    SUM(amount) AS total_revenue,
    COUNT(DISTINCT customer_id) AS unique_customers
FROM catalog.schema.fact_orders
JOIN catalog.schema.dim_store USING (store_id)
GROUP BY order_date, region;
sql
CREATE OR REPLACE MATERIALIZED VIEW catalog.schema.daily_revenue
  CLUSTER BY (order_date)
  SCHEDULE EVERY 1 HOUR
  COMMENT 'Hourly-refreshed daily revenue by region'
AS SELECT
    order_date,
    region,
    SUM(amount) AS total_revenue,
    COUNT(DISTINCT customer_id) AS unique_customers
FROM catalog.schema.fact_orders
JOIN catalog.schema.dim_store USING (store_id)
GROUP BY order_date, region;

Pipe Syntax - Readable Transformations

管道语法 - 可读性更强的转换

sql
-- Traditional SQL rewritten with pipe syntax
FROM catalog.schema.fact_orders
  |> WHERE order_date >= current_date() - INTERVAL 30 DAYS
  |> AGGREGATE SUM(amount) AS total, COUNT(*) AS cnt GROUP BY region, product_category
  |> WHERE total > 10000
  |> ORDER BY total DESC
  |> LIMIT 20;
sql
-- 用管道语法重写传统SQL
FROM catalog.schema.fact_orders
  |> WHERE order_date >= current_date() - INTERVAL 30 DAYS
  |> AGGREGATE SUM(amount) AS total, COUNT(*) AS cnt GROUP BY region, product_category
  |> WHERE total > 10000
  |> ORDER BY total DESC
  |> LIMIT 20;

AI Functions - Enrich Data with LLMs

AI函数 - 用大语言模型丰富数据

sql
-- Classify support tickets
SELECT
  ticket_id,
  description,
  ai_classify(description, ARRAY('billing', 'technical', 'account', 'feature_request')) AS category,
  ai_analyze_sentiment(description) AS sentiment
FROM catalog.schema.support_tickets
LIMIT 100;

-- Extract entities from text
SELECT
  doc_id,
  ai_extract(content, ARRAY('person_name', 'company', 'dollar_amount')) AS entities
FROM catalog.schema.contracts;

-- General-purpose AI query with structured output
SELECT ai_query(
  'databricks-meta-llama-3-3-70b-instruct',
  concat('Summarize this customer feedback in JSON with keys: topic, sentiment, action_items. Feedback: ', feedback),
  returnType => 'STRUCT<topic STRING, sentiment STRING, action_items ARRAY<STRING>>'
) AS analysis
FROM catalog.schema.customer_feedback
LIMIT 50;
sql
-- 分类支持工单
SELECT
  ticket_id,
  description,
  ai_classify(description, ARRAY('billing', 'technical', 'account', 'feature_request')) AS category,
  ai_analyze_sentiment(description) AS sentiment
FROM catalog.schema.support_tickets
LIMIT 100;

-- 从文本中提取实体
SELECT
  doc_id,
  ai_extract(content, ARRAY('person_name', 'company', 'dollar_amount')) AS entities
FROM catalog.schema.contracts;

-- 带结构化输出的通用AI查询
SELECT ai_query(
  'databricks-meta-llama-3-3-70b-instruct',
  concat('Summarize this customer feedback in JSON with keys: topic, sentiment, action_items. Feedback: ', feedback),
  returnType => 'STRUCT<topic STRING, sentiment STRING, action_items ARRAY<STRING>>'
) AS analysis
FROM catalog.schema.customer_feedback
LIMIT 50;

Geospatial - Proximity Search with H3

空间数据 - 基于H3的邻近搜索

sql
-- Find stores within 5km of each customer using H3 indexing
WITH customer_h3 AS (
  SELECT *, h3_longlatash3(longitude, latitude, 7) AS h3_cell
  FROM catalog.schema.customers
),
store_h3 AS (
  SELECT *, h3_longlatash3(longitude, latitude, 7) AS h3_cell
  FROM catalog.schema.stores
)
SELECT
  c.customer_id,
  s.store_id,
  ST_Distance(
    ST_Point(c.longitude, c.latitude),
    ST_Point(s.longitude, s.latitude)
  ) AS distance_m
FROM customer_h3 c
JOIN store_h3 s ON h3_ischildof(c.h3_cell, h3_toparent(s.h3_cell, 5))
WHERE ST_Distance(
  ST_Point(c.longitude, c.latitude),
  ST_Point(s.longitude, s.latitude)
) < 5000;
sql
-- 使用H3索引查找每个客户5公里范围内的门店
WITH customer_h3 AS (
  SELECT *, h3_longlatash3(longitude, latitude, 7) AS h3_cell
  FROM catalog.schema.customers
),
store_h3 AS (
  SELECT *, h3_longlatash3(longitude, latitude, 7) AS h3_cell
  FROM catalog.schema.stores
)
SELECT
  c.customer_id,
  s.store_id,
  ST_Distance(
    ST_Point(c.longitude, c.latitude),
    ST_Point(s.longitude, s.latitude)
  ) AS distance_m
FROM customer_h3 c
JOIN store_h3 s ON h3_ischildof(c.h3_cell, h3_toparent(s.h3_cell, 5))
WHERE ST_Distance(
  ST_Point(c.longitude, c.latitude),
  ST_Point(s.longitude, s.latitude)
) < 5000;

Collation - Case-Insensitive Search

排序规则 - 大小写不敏感搜索

sql
-- Create table with case-insensitive collation
CREATE TABLE catalog.schema.products (
  product_id BIGINT GENERATED ALWAYS AS IDENTITY,
  name STRING COLLATE UTF8_LCASE,
  category STRING COLLATE UTF8_LCASE,
  price DECIMAL(10, 2)
);

-- Queries automatically case-insensitive (no LOWER() needed)
SELECT * FROM catalog.schema.products
WHERE name = 'MacBook Pro';  -- matches 'macbook pro', 'MACBOOK PRO', etc.
sql
-- 创建带大小写不敏感排序规则的表
CREATE TABLE catalog.schema.products (
  product_id BIGINT GENERATED ALWAYS AS IDENTITY,
  name STRING COLLATE UTF8_LCASE,
  category STRING COLLATE UTF8_LCASE,
  price DECIMAL(10, 2)
);

-- 查询自动支持大小写不敏感(无需LOWER())
SELECT * FROM catalog.schema.products
WHERE name = 'MacBook Pro';  -- 匹配'macbook pro'、'MACBOOK PRO'等

http_request - Call External APIs

http_request - 调用外部API

sql
-- Set up connection first (one-time)
CREATE CONNECTION my_api_conn
  TYPE HTTP
  OPTIONS (host 'https://api.example.com', bearer_token secret('scope', 'token'));

-- Call API from SQL
SELECT
  order_id,
  http_request(
    conn => 'my_api_conn',
    method => 'POST',
    path => '/v1/validate',
    json => to_json(named_struct('order_id', order_id, 'amount', amount))
  ).text AS api_response
FROM catalog.schema.orders
WHERE needs_validation = true;
sql
-- 先设置连接(一次性操作)
CREATE CONNECTION my_api_conn
  TYPE HTTP
  OPTIONS (host 'https://api.example.com', bearer_token secret('scope', 'token'));

-- 从SQL中调用API
SELECT
  order_id,
  http_request(
    conn => 'my_api_conn',
    method => 'POST',
    path => '/v1/validate',
    json => to_json(named_struct('order_id', order_id, 'amount', amount))
  ).text AS api_response
FROM catalog.schema.orders
WHERE needs_validation = true;

read_files - Ingest Raw Files

read_files - 导入原始文件

sql
-- Read JSON files from a Volume with schema hints
SELECT *
FROM read_files(
  '/Volumes/catalog/schema/raw/events/',
  format => 'json',
  schemaHints => 'event_id STRING, timestamp TIMESTAMP, payload MAP<STRING, STRING>',
  pathGlobFilter => '*.json',
  recursiveFileLookup => true
);

-- Read CSV with options
SELECT *
FROM read_files(
  '/Volumes/catalog/schema/raw/sales/',
  format => 'csv',
  header => true,
  delimiter => '|',
  dateFormat => 'yyyy-MM-dd',
  schema => 'sale_id INT, sale_date DATE, amount DECIMAL(10,2), store STRING'
);
sql
-- 从Volume读取JSON文件并指定 schema 提示
SELECT *
FROM read_files(
  '/Volumes/catalog/schema/raw/events/',
  format => 'json',
  schemaHints => 'event_id STRING, timestamp TIMESTAMP, payload MAP<STRING, STRING>',
  pathGlobFilter => '*.json',
  recursiveFileLookup => true
);

-- 读取CSV文件并配置参数
SELECT *
FROM read_files(
  '/Volumes/catalog/schema/raw/sales/',
  format => 'csv',
  header => true,
  delimiter => '|',
  dateFormat => 'yyyy-MM-dd',
  schema => 'sale_id INT, sale_date DATE, amount DECIMAL(10,2), store STRING'
);

Recursive CTE - Hierarchy Traversal

递归CTE - 层级遍历

sql
WITH RECURSIVE org_chart AS (
  -- Anchor: top-level managers
  SELECT employee_id, name, manager_id, 0 AS depth, ARRAY(name) AS path
  FROM catalog.schema.employees
  WHERE manager_id IS NULL

  UNION ALL

  -- Recursive: direct reports
  SELECT e.employee_id, e.name, e.manager_id, o.depth + 1, array_append(o.path, e.name)
  FROM catalog.schema.employees e
  JOIN org_chart o ON e.manager_id = o.employee_id
  WHERE o.depth < 10  -- safety limit
)
SELECT * FROM org_chart ORDER BY depth, name;
sql
WITH RECURSIVE org_chart AS (
  -- 锚点:顶层管理者
  SELECT employee_id, name, manager_id, 0 AS depth, ARRAY(name) AS path
  FROM catalog.schema.employees
  WHERE manager_id IS NULL

  UNION ALL

  -- 递归:直接下属
  SELECT e.employee_id, e.name, e.manager_id, o.depth + 1, array_append(o.path, e.name)
  FROM catalog.schema.employees e
  JOIN org_chart o ON e.manager_id = o.employee_id
  WHERE o.depth < 10  -- 安全限制
)
SELECT * FROM org_chart ORDER BY depth, name;

remote_query - Federated Queries

remote_query - 联邦查询

sql
-- Query PostgreSQL via Lakehouse Federation
SELECT *
FROM remote_query(
  'my_postgres_connection',
  database => 'my_database',
  query    => 'SELECT customer_id, email, created_at FROM customers WHERE active = true'
);

sql
-- 通过Lakehouse Federation查询PostgreSQL
SELECT *
FROM remote_query(
  'my_postgres_connection',
  database => 'my_database',
  query    => 'SELECT customer_id, email, created_at FROM customers WHERE active = true'
);

Reference Files

参考文档

Load these for detailed syntax, full parameter lists, and advanced patterns:
FileContentsWhen to Read
references/sql-scripting.mdSQL Scripting, Stored Procedures, Recursive CTEs, TransactionsUser needs procedural SQL, error handling, loops, dynamic SQL
references/materialized-views-pipes.mdMaterialized Views, Temp Tables/Views, Pipe SyntaxUser needs MVs, refresh scheduling, temp objects, pipe operator
references/geospatial-collations.md39 H3 functions, 80+ ST functions, Collation types and hierarchyUser needs spatial analysis, H3 indexing, case/accent handling
references/ai-functions.md13 AI functions, http_request, remote_query, read_files (all options)User needs AI enrichment, API calls, federation, file ingestion
references/best-practices.mdData modeling, performance, Liquid Clustering, anti-patternsUser needs architecture guidance, optimization, or modeling advice

加载以下文档获取详细语法、完整参数列表和高级模式:
文件内容阅读场景
references/sql-scripting.mdSQL脚本、存储过程、递归CTE、事务用户需要过程式SQL、错误处理、循环、动态SQL时
references/materialized-views-pipes.md物化视图、临时表/视图、管道语法用户需要物化视图、刷新调度、临时对象、管道运算符时
references/geospatial-collations.md39个H3函数、80+个ST函数、排序规则类型与层级用户需要空间分析、H3索引、大小写/重音处理时
references/ai-functions.md13个AI函数、http_request、remote_query、read_files(所有参数)用户需要AI增强、API调用、联邦查询、文件导入时
references/best-practices.md数据建模、性能优化、Liquid Clustering、反模式用户需要架构指导、优化建议或建模方案时

Key Guidelines

核心指南

  • Always use Serverless SQL warehouses for AI functions, MVs, and http_request
  • Use
    LIMIT
    during development
    with AI functions to control costs
  • Prefer Liquid Clustering over partitioning for new tables (1-4 keys max)
  • Use
    CLUSTER BY AUTO
    when unsure about clustering keys
  • Star schema in Gold layer for BI; OBT acceptable in Silver
  • Define PK/FK constraints on dimensional models for query optimization
  • Use
    COLLATE UTF8_LCASE
    for user-facing string columns that need case-insensitive search
  • Test SQL via CLI (
    databricks experimental aitools tools query
    ) or notebooks before deploying. If
    --warehouse
    is rejected on your CLI version, set
    DATABRICKS_WAREHOUSE_ID
    in the environment instead.
  • AI函数、物化视图和http_request请始终使用Serverless SQL仓库
  • **开发AI函数时使用
    LIMIT
    **以控制成本
  • 新表优先使用Liquid Clustering而非分区(最多1-4个键)
  • 不确定聚类键时使用
    CLUSTER BY AUTO
  • Gold层使用星型模型用于BI分析;Silver层可使用OBT(宽表)
  • 在维度模型上定义PK/FK约束以优化查询
  • **用户可见的字符串列使用
    COLLATE UTF8_LCASE
    **以支持大小写不敏感搜索
  • 部署前通过CLI(
    databricks experimental aitools tools query
    )或Notebook测试SQL
    。如果你的CLI版本不支持
    --warehouse
    参数,请在环境变量中设置
    DATABRICKS_WAREHOUSE_ID