extension-inference

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Caffeine Inference

LLM extension for Caffeine AI.

面向Caffeine AI的LLM扩展。

Orchestrator routing notes

编排器路由说明

Treat “use an LLM / GPT / chatbot / summarise with AI” as a first-class platform feature. The default path is Caffeine Inference: an OpenAI-compatible chat endpoint that Caffeine hosts, authenticates, and bills for the app. The canister gets its credentials from the platform at runtime; nobody pastes an API key, and the app never stores or returns one.

User intent	Capability
Chat / summarise / classify with an LLM in a Caffeine app	`caffeineai-inference-client` `ChatApi.createChatCompletion` via this skill
Call `api.openai.com` with a user-pasted `sk-...`	`extension-openai` only

Do not load

extension-openai

for a normal Caffeine-app LLM. Do not ask the user for an OpenAI API key. Do not add

setApiKey

endpoints, a key-settings page, or a model picker.

将“使用LLM/GPT/聊天机器人/AI总结”视为一等平台功能。默认路径为Caffeine Inference：这是一个由Caffeine托管、认证并为应用计费的兼容OpenAI的聊天端点。Canister在运行时从平台获取凭证；无需任何人粘贴API密钥，应用也绝不会存储或返回密钥。

用户意图	实现方式
在Caffeine应用中使用LLM进行聊天/总结/分类	通过本技能使用 `caffeineai-inference-client` 的 `ChatApi.createChatCompletion`
使用用户粘贴的 `sk-...` 密钥调用 `api.openai.com`	仅使用 `extension-openai`

请勿为常规Caffeine应用的LLM加载

extension-openai

。请勿向用户索要OpenAI API密钥。请勿添加

setApiKey

端点、密钥设置页面或模型选择器。

Backend

后端

1. Add

caffeineai-inference-client

mops.toml

1. 在

mops.toml

中添加

caffeineai-inference-client

bash

mops add caffeineai-inference-client@0.1.0

Requires Mops ≥ 2.13. Minimum version:

caffeineai-inference-client ≥ 0.1.0

bash

mops add caffeineai-inference-client@0.1.0

要求Mops版本≥2.13。最低版本要求：

caffeineai-inference-client ≥ 0.1.0

。

2. Config comes from the platform

2. 配置来自平台

Config.fromEnv<system>()

returns a complete

Config

— endpoint, bearer, and

is_replicated = ?false

— from the credentials the platform provisions for the app. There is no key to collect and nothing to configure.

Call
```
fromEnv<system>()
```
inside the
```
shared
```
method, or in a
```
<system>
```
-parameterised helper, on every request. A module-level
```
let config = fromEnv
```
will not compile, and a cached
```
Config
```
can go stale when the platform rotates credentials on a running canister.
It traps when the app has no inference credentials. That is a platform condition, not something the app can fix — do not add a "configure AI" empty state or a key-input fallback for it.
Never log the
```
Config
```
, never copy its
```
auth
```
into actor state, and never return it (or any part of it) from a
```
query
```
/
```
shared
```
function.

Config.fromEnv<system>()

会返回完整的

Config

——包含端点、Bearer令牌和

is_replicated = ?false

——这些信息来自平台为应用提供的凭证。无需收集密钥，也无需进行任何配置。

请在
```
shared
```
方法内部，或带有
```
<system>
```
参数的辅助函数中，每次请求时调用
```
fromEnv<system>()
```
。模块级别的
```
let config = fromEnv
```
无法编译，且缓存的
```
Config
```
可能会在平台为运行中的canister轮换凭证时失效。
当应用没有推理凭证时会触发trap。这是平台层面的问题，应用无法修复——请勿添加“配置AI”的空状态或密钥输入回退方案。
绝不要记录
```
Config
```
，绝不要将其
```
auth
```
字段复制到actor状态中，绝不要从
```
query
```
/
```
shared
```
函数返回它（或其任何部分）。

is_replicated = ?false

is REQUIRED

3. 必须设置

is_replicated = ?false

fromEnv

already sets this. Do not override it to

?true

null

Security. A replicated outcall sends the bearer from every replica.
Billing. Replicated outcalls multiply inference spend by subnet size.
Determinism. LLM bodies are sampled; consensus would fail.

fromEnv

已自动设置该值。请勿将其覆盖为

?true

或

null

。

安全性：复制式出站调用会从每个副本发送Bearer令牌。
计费：复制式出站调用会将推理费用乘以子网大小。
确定性：LLM输出是采样生成的；共识会失败。

4. Canonical layout

4. 标准代码结构

motoko

import Inference "lib/inference";

actor {
  public shared func chat(prompt : Text) : async Text {
    await* Inference.runChat<system>(prompt);
  };
};

motoko

import { fromEnv } "mo:caffeineai-inference-client/Config";
import ChatApi "mo:caffeineai-inference-client/Apis/ChatApi";
import ChatCompletionRequest "mo:caffeineai-inference-client/Models/ChatCompletionRequest";
import ChatCompletionRequestMessageOneOf2 "mo:caffeineai-inference-client/Models/ChatCompletionRequestMessageOneOf2";
import Runtime "mo:core/Runtime";

module {
  public func runChat<system>(prompt : Text) : async* Text {
    let config = fromEnv<system>();
    let userMessage = ChatCompletionRequestMessageOneOf2.JSON.init({
      content = #string(prompt);
      role = #user;
    });
    let req = ChatCompletionRequest.JSON.init({
      messages = [#user(userMessage)];
      model = "router";
    });
    let resp = await* ChatApi.createChatCompletion(config, req);
    if (resp.choices.size() == 0) {
      Runtime.trap("Inference returned no choices");
    };
    resp.choices[0].message.content
      ?? Runtime.trap("Inference returned no text content");
  };
};

motoko

import Inference "lib/inference";

actor {
  public shared func chat(prompt : Text) : async Text {
    await* Inference.runChat<system>(prompt);
  };
};

motoko

import { fromEnv } "mo:caffeineai-inference-client/Config";
import ChatApi "mo:caffeineai-inference-client/Apis/ChatApi";
import ChatCompletionRequest "mo:caffeineai-inference-client/Models/ChatCompletionRequest";
import ChatCompletionRequestMessageOneOf2 "mo:caffeineai-inference-client/Models/ChatCompletionRequestMessageOneOf2";
import Runtime "mo:core/Runtime";

module {
  public func runChat<system>(prompt : Text) : async* Text {
    let config = fromEnv<system>();
    let userMessage = ChatCompletionRequestMessageOneOf2.JSON.init({
      content = #string(prompt);
      role = #user;
    });
    let req = ChatCompletionRequest.JSON.init({
      messages = [#user(userMessage)];
      model = "router";
    });
    let resp = await* ChatApi.createChatCompletion(config, req);
    if (resp.choices.size() == 0) {
      Runtime.trap("Inference returned no choices");
    };
    resp.choices[0].message.content
      ?? Runtime.trap("Inference returned no text content");
  };
};

model = "router"

— the platform picks the model

model = "router"

——由平台选择模型

"router"

is the only public model id, and it is a routing tier rather than a model name. Caffeine Inference sizes each request to the complexity of the query — a small fast model for simple prompts, a stronger one for hard reasoning — and reports

"router"

back as the response

model

, so provider names never reach the app.

Always send
```
model = "router"
```
.
Do not add a model dropdown, a "use GPT-4" toggle, or a
```
model
```
parameter on the backend endpoint. There is nothing for the user to choose.
Steer quality with the prompt and with the declared sampling fields (
```
temperature
```
,
```
top_p
```
,
```
max_completion_tokens
```
), not with model selection.

"router"

是唯一的公开模型ID，它是一个路由层而非模型名称。Caffeine Inference会根据查询的复杂度调整每个请求的模型——简单提示使用小型快速模型，复杂推理使用更强大的模型——并在响应中返回

"router"

作为

model

字段，因此应用永远不会接触到供应商的模型名称。

请始终发送
```
model = "router"
```
。
请勿添加模型下拉菜单、“使用GPT-4”切换按钮或后端端点的
```
model
```
参数。用户无需进行任何选择。
通过提示词和声明的采样字段（
```
temperature
```
、
```
top_p
```
、
```
max_completion_tokens
```
）来控制输出质量，而非通过选择模型。

6. Call shapes

6. 调用形式

Function form:

ChatApi.createChatCompletion(config, req) : async*

— use

await*

Suite form:

let api = ChatApi(config); api.createChatCompletion(req) : async

函数形式：

ChatApi.createChatCompletion(config, req) : async*

—— 使用

await*

。

套件形式：

let api = ChatApi(config); api.createChatCompletion(req) : async

。

7. Available API surface — chat completions

7. 可用API范围——聊天补全

caffeineai-inference-client@0.1.0

is generated from

public-api-v0.1.0

Module	Entry point	Route
`ChatApi`	`createChatCompletion`	`POST /v1/chat/completions`
`ModelsApi`	`listModels`	`GET /v1/models` — catalog only; the model is always `"router"` , so an app never needs this

motoko

import ChatApi "mo:caffeineai-inference-client/Apis/ChatApi";
import { fromEnv } "mo:caffeineai-inference-client/Config";

Chat completions are the whole product surface. Not available on this host (404, and not in the package): embeddings, images, audio, moderations, files, legacy completions, Assistants, Responses, and raw

ic.http_request

. If the spec genuinely needs an OpenAI-only API with a pasted

sk-...

, switch to

extension-openai

caffeineai-inference-client@0.1.0

是基于

public-api-v0.1.0

生成的：

模块	入口点	路由
`ChatApi`	`createChatCompletion`	`POST /v1/chat/completions`
`ModelsApi`	`listModels`	`GET /v1/models` —— 仅为目录；模型始终为 `"router"` ，因此应用永远不需要此接口

motoko

import ChatApi "mo:caffeineai-inference-client/Apis/ChatApi";
import { fromEnv } "mo:caffeineai-inference-client/Config";

聊天补全是全部的产品功能范围。在此主机上不可用的功能（会返回404，且未包含在包中）：嵌入、图像、音频、内容审核、文件、旧版补全、Assistants、Responses以及原生

ic.http_request

。如果需求文档确实需要使用粘贴的

sk-...

密钥调用OpenAI专属API，请切换至

extension-openai

。

8. Cycles

8. Cycles设置

defaultConfig.cycles = 30_000_000_000

. Bump for long completions:

motoko

{ fromEnv<system>() with cycles = 100_000_000_000 }

Streaming (

stream = ?true

) is unsupported — management-canister HTTP returns the full body. Leave

stream = null

defaultConfig.cycles = 30_000_000_000

。对于长文本补全，请增加该值：

motoko

{ fromEnv<system>() with cycles = 100_000_000_000 }

流式传输（

stream = ?true

）不受支持——管理canister的HTTP请求会返回完整响应体。请保持

stream = null

。

9. Things that will bite you

9. 需要注意的陷阱

Call
```
fromEnv<system>()
```
inside the
```
shared
```
method (or a
```
<system>
```
helper). A module-level
```
let config = fromEnv
```
will not compile.
```
model = "router"
```
— not
```
"gpt-4o-mini"
```
. See §5.

User turns are

#user(ChatCompletionRequestMessageOneOf2.JSON.init({ content = #string(prompt); role = #user }))

```
JSON.init
```
for required fields; layer optionals with record update. Do not hand-list every
```
null
```
.

resp.choices[0].message.content

?Text

. Check

choices.size()

first.

One chat call is one HTTP outcall inside an update call: budget seconds, not milliseconds.

在
```
shared
```
方法（或
```
<system>
```
辅助函数）内部调用
```
fromEnv<system>()
```
。模块级别的
```
let config = fromEnv
```
无法编译。
```
model = "router"
```
——而非
```
"gpt-4o-mini"
```
。请参见第5节。

用户消息的格式为

#user(ChatCompletionRequestMessageOneOf2.JSON.init({ content = #string(prompt); role = #user }))

。

必填字段使用
```
JSON.init
```
；通过记录更新来处理可选字段。请勿手动列出每个
```
null
```
。

resp.choices[0].message.content

的类型是

?Text

。请先检查

choices.size()

。

一次聊天调用对应更新调用中的一次HTTP出站调用：预算以秒为单位，而非毫秒。

Frontend

前端

The app is ready to chat on first load — there is nothing to configure.

No API-key UI. No settings page, no password input, no "configured?" indicator, no localStorage. If a spec or mock shows an "AI settings" screen, drop it.
No model picker. See §5.
Call the backend chat endpoint (
```
chat(prompt)
```
) and render the returned text. There is no frontend LLM SDK — the canister is the client, so the credentials never reach the browser.
Show a pending state while the call is in flight (an outcall round-trip takes seconds) and surface a retry on trap.

应用在首次加载时即可使用聊天功能——无需任何配置。

无API密钥UI：无设置页面、无密码输入框、无“已配置？”指示器、无本地存储。如果需求文档或原型包含“AI设置”界面，请移除它。
无模型选择器：请参见第5节。
调用后端聊天端点（
```
chat(prompt)
```
）并渲染返回的文本。没有前端LLM SDK——canister作为客户端，因此凭证永远不会到达浏览器。
在调用过程中显示加载状态（出站调用往返需要数秒），并在触发trap时提供重试选项。

extension-inference

Original

Translation

Caffeine Inference

Caffeine Inference

Orchestrator routing notes

编排器路由说明

Backend

后端

1. Add
`caffeineai-inference-client`
to
`mops.toml`

1. 在
`mops.toml`
中添加
`caffeineai-inference-client`

2. Config comes from the platform

2. 配置来自平台

3.
`is_replicated = ?false`
is REQUIRED

3. 必须设置
`is_replicated = ?false`

4. Canonical layout

4. 标准代码结构

5.
`model = "router"`
— the platform picks the model

5.
`model = "router"`
——由平台选择模型

6. Call shapes

6. 调用形式

7. Available API surface — chat completions

7. 可用API范围——聊天补全

8. Cycles

8. Cycles设置

9. Things that will bite you

9. 需要注意的陷阱

Frontend

前端

extension-inference

Original

Translation

Caffeine Inference

Caffeine Inference

Orchestrator routing notes

编排器路由说明

Backend

后端

1. Add caffeineai-inference-client to mops.toml

1. 在mops.toml中添加caffeineai-inference-client

2. Config comes from the platform

2. 配置来自平台

3. is_replicated = ?false is REQUIRED

3. 必须设置is_replicated = ?false

4. Canonical layout

4. 标准代码结构

5. model = "router" — the platform picks the model

5. model = "router"——由平台选择模型

6. Call shapes

6. 调用形式

7. Available API surface — chat completions

7. 可用API范围——聊天补全

8. Cycles

8. Cycles设置

9. Things that will bite you

9. 需要注意的陷阱

Frontend

前端

1. Add
`caffeineai-inference-client`
to
`mops.toml`

1. 在
`mops.toml`
中添加
`caffeineai-inference-client`

3.
`is_replicated = ?false`
is REQUIRED

3. 必须设置
`is_replicated = ?false`

5.
`model = "router"`
— the platform picks the model

5.
`model = "router"`
——由平台选择模型