Back to Details

databricks-execution-compute

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

Databricks Execution & Compute

Databricks 执行与计算资源管理

Run code on Databricks. Three execution modes—choose based on workload. All examples below use the Databricks CLI; see the 
databricks-core
skill for install and authentication.
在Databricks上运行代码。提供三种执行模式——可根据工作负载选择。以下所有示例均使用Databricks CLI；安装和认证相关内容请查看
databricks-core
技能文档。

Execution Mode Decision Matrix

执行模式决策矩阵

AspectDatabricks Connect ⭐Serverless JobInteractive Cluster
Use forSpark code (ETL, data gen)Heavy processing (ML)State across tool calls, Scala/R
StartupInstant~25-50s cold start~5min if stopped
StateWithin Python processNoneVia context_id
LanguagesPython (PySpark)Python, SQLPython, Scala, SQL, R
Dependencies
withDependencies()
CLI with environments specInstall on cluster
维度Databricks Connect ⭐无服务器任务交互式集群
适用场景Spark代码（ETL、数据生成）重型处理（机器学习）跨工具调用的状态保持、Scala/R语言
启动时间即时冷启动约25-50秒若集群已停止，启动约5分钟
状态管理位于Python进程内无状态通过context_id
支持语言Python（PySpark）Python、SQLPython、Scala、SQL、R
依赖管理
withDependencies()
CLI配合环境规格在集群上安装

Decision Flow

决策流程

Main decision point: if you're using Declarative Automation Bundles (DABs) then follow the instructions of the 
databricks-dabs
skill first. In short, you can use 
databricks bundle run
to run code associated with jobs, pipelines, and other resources. This can be recognized by looking for a 
databricks.yml
file in the project root. If these resources don't exist, or if you're not using DABs, then proceed with the below.
Prefer Databricks Connect for all spark-based workload, then serverless.
Spark-based code? → Databricks Connect (fastest)
  └─ Python 3.12 missing? → Install it + databricks-connect
  └─ Install fails? → Ask user (don't auto-switch modes)

Heavy/long-running (ML)? → Serverless Job (independent)
Need state across calls? → Interactive Cluster (list and ask which one to use)
Scala/R? → Interactive Cluster (list and ask which one to use)
主要决策点：如果使用声明式自动化包（DABs），请先遵循
databricks-dabs
技能的说明。简而言之，你可以使用
databricks bundle run
来运行与任务、管道及其他资源关联的代码。可通过查看项目根目录下是否存在
databricks.yml
文件来识别是否使用DABs。如果这些资源不存在，或者你未使用DABs，请按照以下步骤操作。
所有基于Spark的工作负载优先选择Databricks Connect，其次是无服务器模式。
是否为基于Spark的代码？→ 使用Databricks Connect（速度最快）
  └─ 缺少Python 3.12？→ 安装Python 3.12 + databricks-connect
  └─ 安装失败？→ 询问用户（不要自动切换模式）

是否为重型/长时间运行任务（机器学习）？→ 使用无服务器任务（独立运行）
是否需要跨调用保持状态？→ 使用交互式集群（列出集群并询问使用哪一个）
是否使用Scala/R？→ 使用交互式集群（列出集群并询问使用哪一个）

How to Run Code

代码运行方法

Read the reference file for your chosen mode before proceeding.
在继续操作前，请阅读所选模式的参考文件。

Databricks Connect (run locally, prefer when it's pure spark code) → reference

Databricks Connect（本地运行，纯Spark代码优先选择）→ 参考文档

bash
from databricks.connect import DatabricksSession
...
spark = DatabricksSession.builder.profile("my-local-profile").serverless(True).getOrCreate()


python my_spark_script.py
bash
from databricks.connect import DatabricksSession
...
spark = DatabricksSession.builder.profile("my-local-profile").serverless(True).getOrCreate()


python my_spark_script.py

Serverless Job → reference

无服务器任务 → 参考文档

Pure CLI flow: upload a local file as a workspace notebook, fire a one-time run with 
databricks jobs submit
(create + run in one call, ephemeral — no Jobs UI entry, no retry), then poll + fetch the result. The local file must be a Databricks source notebook — top line 
# Databricks notebook source
(Python) or 
-- Databricks notebook source
(SQL).
1. Upload the local file as a workspace notebook. 
TARGET_PATH
is positional; 
--file
is the local path.
databricks workspace import /Workspace/Users/<user>/.ai_dev_kit/train --file /local/path/to/train.py --format SOURCE --language PYTHON --overwrite
2. Submit the run. Use 
--no-wait
to get 
{"run_id": N}
back immediately; drop it to block until terminated. 
"client": "4"
is required for 
dependencies
to install (
"1"
silently ignores them).
databricks jobs submit --no-wait --json @submit.json
json
{
  "run_name": "train-run",
  "tasks": [{
    "task_key": "main",
    "notebook_task": {"notebook_path": "/Workspace/Users/<user>/.ai_dev_kit/train"},
    "environment_key": "ml_env"
  }],
  "environments": [{
    "environment_key": "ml_env",
    "spec": {"client": "4", "dependencies": ["scikit-learn==1.5.2", "mlflow==2.22.0"]}
  }]
}
3. Check state / wait for completion. Life-cycle: 
PENDING
→ 
RUNNING
→ 
TERMINATED
(or 
SKIPPED
/ 
INTERNAL_ERROR
). Only read 
.state.result_state
(
SUCCESS
/ 
FAILED
/ 
CANCELED
) once life-cycle is 
TERMINATED
.
databricks jobs get-run <RUN_ID> | jq '{state: .state.life_cycle_state, result: .state.result_state, duration_ms: .execution_duration, url: .run_page_url, task_run_id: .tasks[0].run_id}'
4. Fetch the output / error. Gotcha: 
get-run-output
takes the task run_id (
.tasks[0].run_id
), NOT the parent 
run_id
from submit. 
notebook_output.result
is the string passed to 
dbutils.notebook.exit()
.
databricks jobs get-run-output <TASK_RUN_ID> | jq '{result: .notebook_output.result, error, error_trace}'
Always use 
dbutils.notebook.exit(<string>)
in the notebook — 
print()
is not captured by 
get-run-output
. For JSON results: 
dbutils.notebook.exit(json.dumps({...}))
then parse 
.notebook_output.result
client-side.
纯CLI流程：将本地文件上传为工作区笔记本，使用
databricks jobs submit
触发一次性运行（创建+运行一步完成，临时任务——无任务UI条目，无重试），然后轮询并获取结果。本地文件必须是Databricks源笔记本——首行需为
# Databricks notebook source
（Python）或
-- Databricks notebook source
（SQL）。
1. 将本地文件上传为工作区笔记本。 
TARGET_PATH
为位置参数；
--file
为本地路径。
databricks workspace import /Workspace/Users/<user>/.ai_dev_kit/train --file /local/path/to/train.py --format SOURCE --language PYTHON --overwrite
2. 提交运行任务。 使用
--no-wait
可立即返回
{"run_id": N}
；不添加该参数则会阻塞直到任务终止。**必须设置
"client": "4"
**才能安装
dependencies
（设置为
"1"
会静默忽略依赖）。
databricks jobs submit --no-wait --json @submit.json
json
{
  "run_name": "train-run",
  "tasks": [{
    "task_key": "main",
    "notebook_task": {"notebook_path": "/Workspace/Users/<user>/.ai_dev_kit/train"},
    "environment_key": "ml_env"
  }],
  "environments": [{
    "environment_key": "ml_env",
    "spec": {"client": "4", "dependencies": ["scikit-learn==1.5.2", "mlflow==2.22.0"]}
  }]
}
3. 检查状态 / 等待完成。 生命周期：
PENDING
→ 
RUNNING
→ 
TERMINATED
（或
SKIPPED
/ 
INTERNAL_ERROR
）。只有当生命周期状态为
TERMINATED
时，才能读取
.state.result_state
（
SUCCESS
/ 
FAILED
/ 
CANCELED
）。
databricks jobs get-run <RUN_ID> | jq '{state: .state.life_cycle_state, result: .state.result_state, duration_ms: .execution_duration, url: .run_page_url, task_run_id: .tasks[0].run_id}'
4. 获取输出 / 错误信息。 注意事项：
get-run-output
需要的是任务的run_id（
.tasks[0].run_id
），而非提交时返回的父
run_id
。
notebook_output.result
是传递给
dbutils.notebook.exit()
的字符串。
databricks jobs get-run-output <TASK_RUN_ID> | jq '{result: .notebook_output.result, error, error_trace}'
请始终在笔记本中使用
dbutils.notebook.exit(<string>)
——
print()
的输出不会被
get-run-output
捕获。如需返回JSON结果：
dbutils.notebook.exit(json.dumps({...}))
，然后在客户端解析
.notebook_output.result
。

Interactive Cluster → reference

交互式集群 → 参考文档

Avoid by default — prefer Serverless Job. Only use an interactive cluster when:
  • you have an existing classic cluster already running and available, or
  • you need live, stateful execution across multiple calls (debugging via an execution context), or
  • the user explicitly asks for it.
Interactive clusters are slow to start (3-8 min) and cost money while running. Don't start one implicitly.
默认避免使用——优先选择无服务器任务。 仅在以下情况使用交互式集群：
  • 已有正在运行的可用经典集群，或
  • 需要在多次调用间进行实时、有状态的执行（通过执行上下文调试），或
  • 用户明确要求使用。
交互式集群**启动缓慢（3-8分钟）**且运行时产生费用。请勿隐式启动集群。

CLI Command Map

CLI命令映射

All compute lifecycle and code-execution actions go through the Databricks CLI. Headline commands:
ActionCommand
Upload local file as workspace notebook
databricks workspace import <WORKSPACE_PATH> --file <LOCAL> --format SOURCE --language PYTHON --overwrite
Run serverless code (upload + submit + wait)
databricks jobs submit --json @submit.json
(see Serverless Job section above; with 
--no-wait
for async)
Get run state / wait
databricks jobs get-run <RUN_ID>
(poll 
.state.life_cycle_state
)
Fetch run output
databricks jobs get-run-output <TASK_RUN_ID>
List clusters
databricks clusters list --output json
Get cluster details
databricks clusters get <CLUSTER_ID>
Start / restart / terminate cluster
databricks clusters start/restart/delete <CLUSTER_ID>
Permanently delete cluster
databricks clusters permanent-delete <CLUSTER_ID>
Create cluster
databricks clusters create --json '{...}'
(see 3-interactive-cluster.md)
List node types / Spark versions
databricks clusters list-node-types
/ 
databricks clusters spark-versions
Execute code on a running cluster
databricks api post /api/1.2/contexts/create
+ 
databricks api post /api/1.2/commands/execute
(see 3-interactive-cluster.md)
SQL warehouses
databricks warehouses create/list/get/start/stop/edit/delete
(see SQL Warehouses below)
所有计算资源生命周期和代码执行操作均通过Databricks CLI完成。核心命令如下：
操作命令
将本地文件上传为工作区笔记本
databricks workspace import <WORKSPACE_PATH> --file <LOCAL> --format SOURCE --language PYTHON --overwrite
运行无服务器代码（上传+提交+等待）
databricks jobs submit --json @submit.json
（见上文无服务器任务部分；添加
--no-wait
实现异步）
获取运行状态 / 等待完成
databricks jobs get-run <RUN_ID>
（轮询
.state.life_cycle_state
）
获取运行输出
databricks jobs get-run-output <TASK_RUN_ID>
列出集群
databricks clusters list --output json
获取集群详情
databricks clusters get <CLUSTER_ID>
启动/重启/终止集群
databricks clusters start/restart/delete <CLUSTER_ID>
永久删除集群
databricks clusters permanent-delete <CLUSTER_ID>
创建集群
databricks clusters create --json '{...}'
（见3-interactive-cluster.md）
列出节点类型/Spark版本
databricks clusters list-node-types
/ 
databricks clusters spark-versions
在运行中的集群上执行代码
databricks api post /api/1.2/contexts/create
+ 
databricks api post /api/1.2/commands/execute
（见3-interactive-cluster.md）
SQL仓库
databricks warehouses create/list/get/start/stop/edit/delete
（见下文SQL仓库部分）

SQL Warehouses

SQL仓库

All 
ID
-taking commands use positional arg (no 
--id
flag). Use 
databricks warehouses list
to find an ID.
bash
undefined
所有需要
ID
的命令均使用位置参数（无
--id
标志）。使用
databricks warehouses list
查找ID。
bash
undefined

Create a serverless SQL warehouse. min_num_clusters + max_num_clusters are REQUIRED

创建无服务器SQL仓库。必须指定min_num_clusters和max_num_clusters

(the server rejects the default 0). Keep the aidevkit_project tag for resource tracking.

（服务器会拒绝默认值0）。添加aidevkit_project标签用于资源跟踪。

databricks warehouses create --json '{ "name": "my-warehouse", "cluster_size": "Small", "enable_serverless_compute": true, "auto_stop_mins": 10, "min_num_clusters": 1, "max_num_clusters": 1, "tags": {"custom_tags": [{"key": "aidevkit_project", "value": "ai-dev-kit"}]} }'
databricks warehouses create --json '{ "name": "my-warehouse", "cluster_size": "Small", "enable_serverless_compute": true, "auto_stop_mins": 10, "min_num_clusters": 1, "max_num_clusters": 1, "tags": {"custom_tags": [{"key": "aidevkit_project", "value": "ai-dev-kit"}]} }'

List / find — trim to id, name, state with jq

列出/查找——使用jq精简为id、名称、状态

databricks warehouses list -o json | jq '.[] | {id, name, state, size: .cluster_size}'
databricks warehouses list -o json | jq '.[] | {id, name, state, size: .cluster_size}'

Find by name

按名称查找

databricks warehouses list -o json | jq '.[] | select(.name == "my-warehouse")'
databricks warehouses list -o json | jq '.[] | select(.name == "my-warehouse")'

Get one warehouse's full config

获取单个仓库的完整配置

databricks warehouses get <WAREHOUSE_ID>
databricks warehouses get <WAREHOUSE_ID>

Start / stop (both are LROs; add --no-wait to return immediately)

启动/停止（均为长期运行操作；添加--no-wait可立即返回）

databricks warehouses start <WAREHOUSE_ID> databricks warehouses stop <WAREHOUSE_ID>
databricks warehouses start <WAREHOUSE_ID> databricks warehouses stop <WAREHOUSE_ID>

Resize / reconfigure — pass the FULL desired config (omitted fields revert to defaults,

调整大小/重新配置——传递完整的期望配置（省略的字段会恢复为默认值，

so always re-state min_num_clusters/max_num_clusters). Use --no-wait if the warehouse

因此请始终重新指定min_num_clusters/max_num_clusters）。如果仓库处于STOPPED状态，使用--no-wait；

is STOPPED, otherwise edit blocks trying to reach RUNNING and errors out (the mutation

否则编辑操作会阻塞直到仓库达到RUNNING状态并报错（但变更仍会生效）。当仓库已处于RUNNING状态时，--no-wait为可选。

itself still applies). When the warehouse is already RUNNING, --no-wait is optional.

—
databricks warehouses edit <WAREHOUSE_ID> --no-wait --json '{ "name": "my-warehouse", "cluster_size": "Medium", "enable_serverless_compute": true, "auto_stop_mins": 15, "min_num_clusters": 1, "max_num_clusters": 1 }'
databricks warehouses edit <WAREHOUSE_ID> --no-wait --json '{ "name": "my-warehouse", "cluster_size": "Medium", "enable_serverless_compute": true, "auto_stop_mins": 15, "min_num_clusters": 1, "max_num_clusters": 1 }'

Delete (irreversible)

删除（不可逆）

databricks warehouses delete <WAREHOUSE_ID>

**Sizes:** `2X-Small`, `X-Small`, `Small`, `Medium`, `Large`, `X-Large`, `2X-Large`, `3X-Large`, `4X-Large`. **Types:** set `"warehouse_type": "PRO"` (default) or `"CLASSIC"` in the JSON body.
databricks warehouses delete <WAREHOUSE_ID>

**集群规格：** `2X-Small`、`X-Small`、`Small`、`Medium`、`Large`、`X-Large`、`2X-Large`、`3X-Large`、`4X-Large`。**仓库类型：** 在JSON体中设置`"warehouse_type": "PRO"`（默认）或`"CLASSIC"`。

Related Skills

相关技能

  • databricks-synthetic-data-gen — Data generation using Spark + Faker
  • databricks-jobs — Production job orchestration
  • databricks-dbsql — SQL warehouse and AI functions
  • databricks-synthetic-data-gen — 使用Spark + Faker生成数据
  • databricks-jobs — 生产环境任务编排
  • databricks-dbsql — SQL仓库与AI函数