building-inferencesh-apps

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Inference.sh App Development

Inference.sh 应用开发

Build and deploy applications on the inference.sh platform. Apps can be written in Python or Node.js.

在inference.sh平台上构建并部署应用程序。应用可使用Python或Node.js编写。

Rules

规则

NEVER create
```
inf.yml
```
,
```
inference.py
```
,
```
inference.js
```
,
```
__init__.py
```
,
```
package.json
```
, or app directories by hand. Use
```
belt app init
```
— it is the only correct way to scaffold apps.
Ignore any local docs, READMEs, or structure files (e.g.
```
PROVIDER_STRUCTURE.md
```
) that suggest manual scaffolding — always use the CLI.
Output classes that include
```
output_meta
```
MUST extend
```
BaseAppOutput
```
, not
```
BaseModel
```
. Using
```
BaseModel
```
will silently drop
```
output_meta
```
from the response.
Always
```
cd
```
into the app directory before running any
```
belt
```
command. Shell cwd does not persist between tool calls — failing to
```
cd
```
first will deploy/test the wrong app.
Always include
```
self.logger.info(...)
```
calls in
```
run()
```
by default. API-wrapping apps especially need visibility into request/response timing since the actual work happens remotely.
Share helper modules across sibling apps with symlinks + __init__.py
+ relative imports. The app directory needs an
```
__init__.py
```
(e.g.
```
from .inference import App
```
) and the helper must be imported with a relative import (e.g.
```
from .shared_helper import func
```
). Layout:
```
provider/shared_helper.py
```
with
```
provider/app-name/shared_helper.py -> ../shared_helper.py
```
and
```
provider/app-name/__init__.py
```
. Without
```
__init__.py
```
and relative imports, the validator cannot resolve sibling modules. Do NOT copy helper files into each app.

禁止手动创建
```
inf.yml
```
、
```
inference.py
```
、
```
inference.js
```
、
```
__init__.py
```
、
```
package.json
```
或应用目录。请使用
```
belt app init
```
命令——这是搭建应用的唯一正确方式。
忽略任何建议手动搭建的本地文档、README或结构文件（如
```
PROVIDER_STRUCTURE.md
```
）——始终使用CLI工具。
包含
```
output_meta
```
的输出类必须继承
```
BaseAppOutput
```
，而非
```
BaseModel
```
。使用
```
BaseModel
```
会导致
```
output_meta
```
从响应中被静默丢弃。
在运行任何
```
belt
```
命令前，务必先
```
cd
```
进入应用目录。Shell的当前工作目录不会在工具调用之间持久化——若不先执行
```
cd
```
，可能会部署/测试错误的应用。
默认情况下，
```
run()
```
方法中必须包含
```
self.logger.info(...)
```
调用。尤其是封装API的应用，由于实际工作在远程执行，需要可见的请求/响应计时信息。
通过符号链接 + __init__.py
+ 相对导入在同级应用间共享辅助模块。应用目录需要包含
```
__init__.py
```
（例如
```
from .inference import App
```
），且辅助模块必须通过相对导入引入（例如
```
from .shared_helper import func
```
）。目录结构示例：
```
provider/shared_helper.py
```
，同时在
```
provider/app-name/
```
下创建
```
shared_helper.py -> ../shared_helper.py
```
的符号链接和
```
__init__.py
```
。若缺少
```
__init__.py
```
和相对导入，验证器将无法解析同级模块。请勿将辅助文件复制到每个应用中。

CLI Installation

CLI 安装

bash

curl -fsSL https://cli.inference.sh | sh

bash

belt update   # Update CLI
belt login    # Authenticate
belt me       # Check current user

bash

curl -fsSL https://cli.inference.sh | sh

bash

belt update   # 更新CLI
belt login    # 身份验证
belt me       # 查看当前用户

Quick Start

快速开始

Scaffold new apps with

belt app init

(see Rules above). It generates the correct project structure,

inf.yml

, and boilerplate — avoiding common mistakes like missing

"type": "module"

package.json

or incorrect kernel names.

bash

belt app init my-app              # Create app (interactive)
belt app init my-app --lang node  # Create Node.js app

使用

belt app init

搭建新应用（请遵循上述规则）。该命令会生成正确的项目结构、

inf.yml

文件和模板代码——避免常见错误，比如

package.json

中缺少

"type": "module"

或内核名称不正确。

bash

belt app init my-app              # 创建应用（交互式）
belt app init my-app --lang node  # 创建Node.js应用

Development Workflow (mandatory)

开发流程（强制要求）

Every app MUST go through this full cycle. Do not skip steps.

每个应用都必须完成完整的流程，请勿跳过步骤。

1. Scaffold

1. 搭建项目

bash

belt app init my-app

bash

belt app init my-app

2. Implement

2. 实现功能

Write

inference.py

(or

inference.js

inf.yml

, and

requirements.txt

(or

package.json

编写

inference.py

（或

inference.js

）、

inf.yml

和

requirements.txt

（或

package.json

）。

3. Test Locally

3. 本地测试

bash

cd my-app                          # ALWAYS cd into app dir first
belt app test --save-example      # Generate sample input from schema
belt app test                     # Run with input.json
belt app test --input '{"prompt": "hello"}'  # Or inline JSON

bash

cd my-app                          # 务必先进入应用目录
belt app test --save-example      # 根据Schema生成示例输入
belt app test                     # 使用input.json运行测试
belt app test --input '{"prompt": "hello"}'  # 或使用内联JSON

4. Deploy

4. 部署

bash

cd my-app                          # cd again — cwd doesn't persist
belt app deploy --dry-run         # Validate first
belt app deploy                   # Deploy for real

bash

cd my-app                          # 再次进入目录——当前工作目录不会持久化
belt app deploy --dry-run         # 先验证配置
belt app deploy                   # 正式部署

5. Cloud Test & Verify

5. 云端测试与验证

After deploying, test the live version and verify

output_meta

is present in the response:

bash

belt app run user/app --json --input '{"prompt": "hello"}'

Check the JSON response for

output_meta

— if it's missing, the output class is likely extending

BaseModel

instead of

BaseAppOutput

bash

undefined

部署完成后，测试线上版本并验证响应中是否包含

output_meta

：

bash

belt app run user/app --json --input '{"prompt": "hello"}'

检查JSON响应中的

output_meta

——如果缺失，说明输出类可能继承了

BaseModel

而非

BaseAppOutput

。

bash

undefined

Other useful commands

其他实用命令

belt app run user/app --input input.json belt app sample user/app belt app sample user/app --save input.json

undefined

belt app run user/app --input input.json belt app sample user/app belt app sample user/app --save input.json

undefined

App Structure

应用结构

Python

python

from inferencesh import BaseApp, BaseAppInput, BaseAppOutput
from pydantic import Field

class AppSetup(BaseAppInput):
    """Setup parameters — triggers re-init when changed"""
    model_id: str = Field(default="gpt2", description="Model to load")

class AppInput(BaseAppInput):
    prompt: str = Field(description="Input prompt")

class AppOutput(BaseAppOutput):
    result: str = Field(description="Output result")

class App(BaseApp):
    async def setup(self, config: AppSetup):
        """Runs once when worker starts or config changes"""
        self.model = load_model(config.model_id)

    async def run(self, input_data: AppInput) -> AppOutput:
        """Default function — runs for each request"""
        self.logger.info(f"Processing prompt: {input_data.prompt[:50]}")
        result = self.model.generate(input_data.prompt)
        self.logger.info("Generation complete")
        return AppOutput(result=result)

    async def unload(self):
        """Cleanup on shutdown"""
        pass

    async def on_cancel(self):
        """Called when user cancels — for long-running tasks"""
        return True

python

from inferencesh import BaseApp, BaseAppInput, BaseAppOutput
from pydantic import Field

class AppSetup(BaseAppInput):
    """Setup parameters — triggers re-init when changed"""
    model_id: str = Field(default="gpt2", description="Model to load")

class AppInput(BaseAppInput):
    prompt: str = Field(description="Input prompt")

class AppOutput(BaseAppOutput):
    result: str = Field(description="Output result")

class App(BaseApp):
    async def setup(self, config: AppSetup):
        """Runs once when worker starts or config changes"""
        self.model = load_model(config.model_id)

    async def run(self, input_data: AppInput) -> AppOutput:
        """Default function — runs for each request"""
        self.logger.info(f"Processing prompt: {input_data.prompt[:50]}")
        result = self.model.generate(input_data.prompt)
        self.logger.info("Generation complete")
        return AppOutput(result=result)

    async def unload(self):
        """Cleanup on shutdown"""
        pass

    async def on_cancel(self):
        """Called when user cancels — for long-running tasks"""
        return True

Node.js

javascript

import { z } from "zod";

export const AppSetup = z.object({
  modelId: z.string().default("gpt2").describe("Model to load"),
});

export const RunInput = z.object({
  prompt: z.string().describe("Input prompt"),
});

export const RunOutput = z.object({
  result: z.string().describe("Output result"),
});

export class App {
  async setup(config) {
    /** Runs once when worker starts or config changes */
    this.model = loadModel(config.modelId);
  }

  async run(inputData) {
    /** Default function — runs for each request */
    return { result: "done" };
  }

  async unload() {
    /** Cleanup on shutdown */
  }

  async onCancel() {
    /** Called when user cancels — for long-running tasks */
    return true;
  }
}

javascript

import { z } from "zod";

export const AppSetup = z.object({
  modelId: z.string().default("gpt2").describe("Model to load"),
});

export const RunInput = z.object({
  prompt: z.string().describe("Input prompt"),
});

export const RunOutput = z.object({
  result: z.string().describe("Output result"),
});

export class App {
  async setup(config) {
    /** Runs once when worker starts or config changes */
    this.model = loadModel(config.modelId);
  }

  async run(inputData) {
    /** Default function — runs for each request */
    return { result: "done" };
  }

  async unload() {
    /** Cleanup on shutdown */
  }

  async onCancel() {
    /** Called when user cancels — for long-running tasks */
    return true;
  }
}

Multi-Function Apps

多功能应用

Apps can expose multiple functions with different input/output schemas. Functions are auto-discovered.

Python: Add methods with type-hinted Pydantic input/output models. Node.js: Export

{PascalName}Input

and

{PascalName}Output

Zod schemas for each method.

Functions must be public (no

prefix) and not lifecycle methods (

setup

unload

on_cancel

onCancel

constructor

Call via API with

"function": "method_name"

in the request body. Set

default_function

inf.yml

to change which function is called when none is specified (defaults to

run

应用可以暴露多个具有不同输入/输出Schema的函数，函数会被自动发现。

Python： 添加带有类型提示的Pydantic输入/输出模型的方法。 Node.js： 为每个方法导出

{PascalName}Input

和

{PascalName}Output

Zod Schema。

函数必须是公共方法（不能以

开头），且不能是生命周期方法（

setup

、

unload

、

on_cancel

onCancel

、

constructor

）。

通过API调用时，在请求体中指定

"function": "method_name"

。可在

inf.yml

中设置

default_function

，修改未指定函数时的默认调用方法（默认为

run

）。

API-Wrapper App Template (Python)

API封装应用模板（Python）

Most CPU-only apps that wrap external APIs follow this pattern. Use this as a starting point:

python

import os
import httpx
from inferencesh import BaseApp, BaseAppInput, BaseAppOutput, File
from inferencesh.models.usage import OutputMeta, ImageMeta  # or TextMeta, AudioMeta, etc.
from pydantic import Field

class AppInput(BaseAppInput):
    prompt: str = Field(description="Input prompt")

class AppOutput(BaseAppOutput):  # NOT BaseModel — output_meta requires this
    image: File = Field(description="Generated image")

class App(BaseApp):
    async def setup(self, config):
        self.api_key = os.environ["API_KEY"]
        self.client = httpx.AsyncClient(timeout=120)

    async def run(self, input_data: AppInput) -> AppOutput:
        self.logger.info(f"Calling API with prompt: {input_data.prompt[:80]}")

        response = await self.client.post(
            "https://api.example.com/generate",
            headers={"Authorization": f"Bearer {self.api_key}"},
            json={"prompt": input_data.prompt},
        )
        response.raise_for_status()

        # Write output file
        output_path = "/tmp/output.png"
        with open(output_path, "wb") as f:
            f.write(response.content)

        # Read actual dimensions (don't hardcode!)
        from PIL import Image
        with Image.open(output_path) as img:
            width, height = img.size

        self.logger.info(f"Generated {width}x{height} image")

        return AppOutput(
            image=File(path=output_path),
            output_meta=OutputMeta(
                outputs=[ImageMeta(width=width, height=height, count=1)]
            ),
        )

    async def unload(self):
        await self.client.aclose()

大多数仅使用CPU、封装外部API的应用遵循此模式，可将其作为起点：

python

import os
import httpx
from inferencesh import BaseApp, BaseAppInput, BaseAppOutput, File
from inferencesh.models.usage import OutputMeta, ImageMeta  # or TextMeta, AudioMeta, etc.
from pydantic import Field

class AppInput(BaseAppInput):
    prompt: str = Field(description="Input prompt")

class AppOutput(BaseAppOutput):  # NOT BaseModel — output_meta requires this
    image: File = Field(description="Generated image")

class App(BaseApp):
    async def setup(self, config):
        self.api_key = os.environ["API_KEY"]
        self.client = httpx.AsyncClient(timeout=120)

    async def run(self, input_data: AppInput) -> AppOutput:
        self.logger.info(f"Calling API with prompt: {input_data.prompt[:80]}")

        response = await self.client.post(
            "https://api.example.com/generate",
            headers={"Authorization": f"Bearer {self.api_key}"},
            json={"prompt": input_data.prompt},
        )
        response.raise_for_status()

        # Write output file
        output_path = "/tmp/output.png"
        with open(output_path, "wb") as f:
            f.write(response.content)

        # Read actual dimensions (don't hardcode!)
        from PIL import Image
        with Image.open(output_path) as img:
            width, height = img.size

        self.logger.info(f"Generated {width}x{height} image")

        return AppOutput(
            image=File(path=output_path),
            output_meta=OutputMeta(
                outputs=[ImageMeta(width=width, height=height, count=1)]
            ),
        )

    async def unload(self):
        await self.client.aclose()

Configuring Resources (inf.yml)

配置资源（inf.yml）

Project Structure

项目结构

Python:

my-app/
├── inf.yml           # Configuration
├── inference.py      # App logic
├── requirements.txt  # Python packages (pip)
└── packages.txt      # System packages (apt) — optional

Node.js:

my-app/
├── inf.yml           # Configuration
├── src/
│   └── inference.js  # App logic
├── package.json      # Node.js packages (npm/pnpm)
└── packages.txt      # System packages (apt) — optional

Python：

my-app/
├── inf.yml           # 配置文件
├── inference.py      # 应用逻辑
├── requirements.txt  # Python依赖包（pip）
└── packages.txt      # 系统依赖包（apt）——可选

Node.js：

my-app/
├── inf.yml           # 配置文件
├── src/
│   └── inference.js  # 应用逻辑
├── package.json      # Node.js依赖包（npm/pnpm）
└── packages.txt      # 系统依赖包（apt）——可选

inf.yml

yaml

name: my-app
description: What my app does
category: image
kernel: python-3.11     # or node-22

yaml

name: my-app
description: What my app does
category: image
kernel: python-3.11     # 或 node-22

For multi-function apps (default: run)

多功能应用配置（默认：run）

default_function: generate

resources: gpu: count: 1 vram: 24 # 24GB (auto-converted) type: any ram: 32 # 32GB

env: MODEL_NAME: gpt-4

secrets:

key: HF_TOKEN description: HuggingFace token for gated models optional: false

integrations:

key: google.sheets description: Access to Google Sheets optional: true

undefined

resources: gpu: count: 1 vram: 24 # 24GB（自动转换单位） type: any ram: 32 # 32GB

env: MODEL_NAME: gpt-4

secrets:

key: HF_TOKEN description: HuggingFace token for gated models optional: false

integrations:

key: google.sheets description: Access to Google Sheets optional: true

undefined

Resource Units

资源单位

CLI auto-converts human-friendly values:

< 1000 → GB (e.g.,
```
80
```
= 80GB)
1000 to 1B → MB

CLI会自动转换人性化的数值：

小于1000 → GB（例如
```
80
```
= 80GB）
1000到10亿 → MB

GPU Types

GPU类型

any

nvidia

amd

apple

none

Note: Currently only NVIDIA CUDA GPUs are supported.

any

nvidia

amd

apple

none

注意： 当前仅支持NVIDIA CUDA GPU。

应用分类

image

video

audio

text

chat

3d

other

image

video

audio

text

chat

3d

other

CPU-Only Apps

仅CPU应用

yaml

resources:
  gpu:
    count: 0
    type: none
  ram: 4

yaml

resources:
  gpu:
    count: 0
    type: none
  ram: 4

Dependencies

依赖配置

Python —

requirements.txt

torch>=2.0
transformers
accelerate

Node.js —

package.json

json

{
  "type": "module",
  "dependencies": {
    "zod": "^3.23.0",
    "sharp": "^0.33.0"
  }
}

System packages —

packages.txt

(apt-installable):

ffmpeg
libgl1-mesa-glx

Python —

requirements.txt

torch>=2.0
transformers
accelerate

Node.js —

package.json

json

{
  "type": "module",
  "dependencies": {
    "zod": "^3.23.0",
    "sharp": "^0.33.0"
  }
}

系统依赖包 —

packages.txt

（可通过apt安装）：

ffmpeg
libgl1-mesa-glx

Base Images

基础镜像

Type	Image
GPU	`docker.inference.sh/gpu:latest-cuda`
CPU	`docker.inference.sh/cpu:latest`

类型	镜像
GPU	`docker.inference.sh/gpu:latest-cuda`
CPU	`docker.inference.sh/cpu:latest`

GPU Apps

GPU应用

Always use
accelerate
for device detection —

torch.cuda.is_available()

doesn't reliably detect GPUs in grid containers:

python

from accelerate import Accelerator

accelerator = Accelerator()
self.device = accelerator.device

Always
.to(device)
explicitly — don't rely on

device_map

kwargs, they silently fall back to CPU if the library doesn't support them:

python

self.model = SomeModel.from_pretrained("org/model")
self.model = self.model.to(device=self.device, dtype=torch.float16)

Remember to add

accelerate

requirements.txt

始终使用
accelerate
进行设备检测——

torch.cuda.is_available()

无法在网格容器中可靠检测GPU：

python

from accelerate import Accelerator

accelerator = Accelerator()
self.device = accelerator.device

始终显式调用
.to(device)
——不要依赖

device_map

参数，若库不支持该参数，会静默回退到CPU：

python

self.model = SomeModel.from_pretrained("org/model")
self.model = self.model.to(device=self.device, dtype=torch.float16)

记得将

accelerate

添加到

requirements.txt

中。

Reference Files

参考文件

Load the appropriate reference file based on the language and topic:

根据语言和主题加载对应的参考文件：

App Logic & Schemas

应用逻辑与Schema

references/python-app-logic.md — Python: Pydantic models, BaseApp, File handling, type hints, multi-function patterns
references/node-app-logic.md — Node.js: Zod schemas, File handling, ESM, generators, multi-function patterns

references/python-app-logic.md — Python：Pydantic模型、BaseApp、文件处理、类型提示、多功能模式
references/node-app-logic.md — Node.js：Zod Schema、文件处理、ESM、生成器、多功能模式

Debugging, Optimization & Cancellation

调试、优化与取消

references/python-patterns.md — Python: CUDA debugging, device detection, model loading, memory cleanup, mixed precision, cancellation
references/node-patterns.md — Node.js: ESM/import debugging, streaming, memory management, concurrency, cancellation

references/python-patterns.md — Python：CUDA调试、设备检测、模型加载、内存清理、混合精度、取消操作
references/node-patterns.md — Node.js：ESM/导入调试、流处理、内存管理、并发、取消操作

Secrets & OAuth

密钥与OAuth

references/python-secrets-oauth.md — Python: os.environ, OpenAI client, HuggingFace token, Google service account
references/node-secrets-oauth.md — Node.js: process.env, OpenAI client, Google credentials JSON

references/python-secrets-oauth.md — Python：os.environ、OpenAI客户端、HuggingFace令牌、Google服务账号
references/node-secrets-oauth.md — Node.js：process.env、OpenAI客户端、Google凭证JSON

Usage Tracking

使用追踪

references/python-tracking.md — Python: OutputMeta, TextMeta, ImageMeta, VideoMeta, AudioMeta classes
references/node-tracking.md — Node.js: textMeta, imageMeta, videoMeta, audioMeta factory functions

references/python-tracking.md — Python：OutputMeta、TextMeta、ImageMeta、VideoMeta、AudioMeta类
references/node-tracking.md — Node.js：textMeta、imageMeta、videoMeta、audioMeta工厂函数

CLI

references/cli.md — Full CLI command reference, prerequisites for both languages

references/cli.md — 完整CLI命令参考、两种语言的前置要求

Resources

资源链接

Full Docs: inference.sh/docs
Examples: github.com/inference-sh/grid

完整文档：inference.sh/docs
示例代码：github.com/inference-sh/grid

building-inferencesh-apps

Original

Translation

Inference.sh App Development

Inference.sh 应用开发

Rules

规则

CLI Installation

CLI 安装

Quick Start

快速开始

Development Workflow (mandatory)

开发流程（强制要求）

1. Scaffold

1. 搭建项目

2. Implement

2. 实现功能

3. Test Locally

3. 本地测试

4. Deploy

4. 部署

5. Cloud Test & Verify

5. 云端测试与验证

Other useful commands

其他实用命令

App Structure

应用结构

Python

Python

Node.js

Node.js

Multi-Function Apps

多功能应用

API-Wrapper App Template (Python)

API封装应用模板（Python）

Configuring Resources (inf.yml)

配置资源（inf.yml）

Project Structure

项目结构

inf.yml

inf.yml

For multi-function apps (default: run)

多功能应用配置（默认：run）

default_function: generate

default_function: generate

Resource Units

资源单位

GPU Types

GPU类型

Categories

应用分类

CPU-Only Apps

仅CPU应用

Dependencies

依赖配置

Base Images

基础镜像

GPU Apps

GPU应用

Reference Files

参考文件

App Logic & Schemas

应用逻辑与Schema

Debugging, Optimization & Cancellation

调试、优化与取消

Secrets & OAuth

密钥与OAuth

Usage Tracking

使用追踪

CLI

CLI

Resources

资源链接