Ollama Stack

Ollama 本地LLM栈

Deploy a local LLM stack for offline and privacy-first workflows.

为离线和隐私优先的工作流部署本地LLM栈。

When to Use This Skill

适用场景

Use this skill when:

Setting up private/local LLM inference for development
Building air-gapped AI environments
Running models on personal hardware (Mac, Linux, Windows with GPU)
Creating team-shared inference endpoints
Prototyping before committing to cloud LLM APIs

在以下场景中使用本技能：

为开发搭建私有/本地LLM推理服务
构建离线AI环境
在个人硬件（Mac、Linux、带GPU的Windows）上运行模型
创建团队共享的推理端点
在接入云LLM API之前进行原型开发

Prerequisites

前置条件

8 GB+ RAM (16 GB+ recommended for 7B+ models)
For GPU acceleration: NVIDIA GPU with 6 GB+ VRAM, or Apple Silicon Mac
Docker (for containerized deployment)
20 GB+ disk for model storage

8GB及以上内存（7B参数以上模型建议16GB+）
如需GPU加速：NVIDIA GPU（6GB+显存）或Apple Silicon Mac
Docker（用于容器化部署）
20GB+磁盘空间用于模型存储

Quick Start

快速开始

bash

undefined

bash

undefined

Install Ollama

curl -fsSL https://ollama.com/install.sh | sh

Start the server

ollama serve

Pull and run a model

ollama pull llama3.1:8b ollama run llama3.1:8b "Explain Kubernetes pods in one paragraph"

List available models

ollama list

Pull specific quantization

ollama pull llama3.1:8b-instruct-q4_K_M

undefined

ollama pull llama3.1:8b-instruct-q4_K_M

undefined

Model Selection Guide

模型选择指南

Model	Size	VRAM	Best For
`llama3.1:8b`	4.7 GB	6 GB	General chat, coding
`llama3.1:70b`	40 GB	48 GB	Complex reasoning
`codellama:13b`	7.4 GB	10 GB	Code generation
`mistral:7b`	4.1 GB	6 GB	Fast general tasks
`mixtral:8x7b`	26 GB	32 GB	High-quality MoE
`nomic-embed-text`	274 MB	1 GB	Embeddings for RAG
`llava:13b`	8 GB	10 GB	Vision + text
`deepseek-coder-v2:16b`	9 GB	12 GB	Code generation
`qwen2.5:14b`	9 GB	12 GB	Multilingual, reasoning

模型	大小	显存要求	最适用场景
`llama3.1:8b`	4.7 GB	6 GB	通用对话、代码编写
`llama3.1:70b`	40 GB	48 GB	复杂推理
`codellama:13b`	7.4 GB	10 GB	代码生成
`mistral:7b`	4.1 GB	6 GB	快速通用任务
`mixtral:8x7b`	26 GB	32 GB	高质量MoE模型
`nomic-embed-text`	274 MB	1 GB	RAG场景嵌入向量生成
`llava:13b`	8 GB	10 GB	视觉+文本任务
`deepseek-coder-v2:16b`	9 GB	12 GB	代码生成
`qwen2.5:14b`	9 GB	12 GB	多语言处理、推理

Docker Compose — Full Stack

Docker Compose — 完整部署栈

yaml

undefined

yaml

undefined

docker-compose.yml

services: ollama: image: ollama/ollama:latest container_name: ollama restart: unless-stopped ports: - "11434:11434" volumes: - ollama_data:/root/.ollama environment: - OLLAMA_HOST=0.0.0.0 - OLLAMA_NUM_PARALLEL=4 - OLLAMA_MAX_LOADED_MODELS=2 - OLLAMA_FLASH_ATTENTION=1 deploy: resources: reservations: devices: - driver: nvidia count: all capabilities: [gpu] healthcheck: test: ["CMD", "curl", "-f", "http://localhost:11434/api/tags"] interval: 30s timeout: 10s retries: 3

open-webui: image: ghcr.io/open-webui/open-webui:main container_name: open-webui restart: unless-stopped ports: - "3000:8080" volumes: - webui_data:/app/backend/data environment: - OLLAMA_BASE_URL=http://ollama:11434 - WEBUI_AUTH=true - WEBUI_SECRET_KEY=${WEBUI_SECRET_KEY:-change-me-in-production} - DEFAULT_MODELS=llama3.1:8b depends_on: ollama: condition: service_healthy

litellm: image: ghcr.io/berriai/litellm:main-latest container_name: litellm restart: unless-stopped ports: - "4000:4000" volumes: - ./litellm-config.yaml:/app/config.yaml command: ["--config", "/app/config.yaml"] depends_on: ollama: condition: service_healthy

volumes: ollama_data: webui_data:

undefined

services: ollama: image: ollama/ollama:latest container_name: ollama restart: unless-stopped ports: - "11434:11434" volumes: - ollama_data:/root/.ollama environment: - OLLAMA_HOST=0.0.0.0 - OLLAMA_NUM_PARALLEL=4 - OLLAMA_MAX_LOADED_MODELS=2 - OLLAMA_FLASH_ATTENTION=1 deploy: resources: reservations: devices: - driver: nvidia count: all capabilities: [gpu] healthcheck: test: ["CMD", "curl", "-f", "http://localhost:11434/api/tags"] interval: 30s timeout: 10s retries: 3

open-webui: image: ghcr.io/open-webui/open-webui:main container_name: open-webui restart: unless-stopped ports: - "3000:8080" volumes: - webui_data:/app/backend/data environment: - OLLAMA_BASE_URL=http://ollama:11434 - WEBUI_AUTH=true - WEBUI_SECRET_KEY=${WEBUI_SECRET_KEY:-change-me-in-production} - DEFAULT_MODELS=llama3.1:8b depends_on: ollama: condition: service_healthy

litellm: image: ghcr.io/berriai/litellm:main-latest container_name: litellm restart: unless-stopped ports: - "4000:4000" volumes: - ./litellm-config.yaml:/app/config.yaml command: ["--config", "/app/config.yaml"] depends_on: ollama: condition: service_healthy

volumes: ollama_data: webui_data:

undefined

LiteLLM Proxy Config

LiteLLM 代理配置

yaml

undefined

yaml

undefined

litellm-config.yaml

model_list:

model_name: llama3 litellm_params: model: ollama/llama3.1:8b api_base: http://ollama:11434
model_name: codellama litellm_params: model: ollama/codellama:13b api_base: http://ollama:11434
model_name: embeddings litellm_params: model: ollama/nomic-embed-text api_base: http://ollama:11434

general_settings: master_key: sk-local-dev-key max_budget: 0 # unlimited for local

undefined

model_list:

model_name: llama3 litellm_params: model: ollama/llama3.1:8b api_base: http://ollama:11434
model_name: codellama litellm_params: model: ollama/codellama:13b api_base: http://ollama:11434
model_name: embeddings litellm_params: model: ollama/nomic-embed-text api_base: http://ollama:11434

general_settings: master_key: sk-local-dev-key max_budget: 0 # unlimited for local

undefined

API Usage

API 使用方法

Ollama exposes an OpenAI-compatible API:

bash

undefined

Ollama 提供兼容OpenAI的API：

bash

undefined

Chat completion

curl http://localhost:11434/v1/chat/completions
-H "Content-Type: application/json"
-d '{ "model": "llama3.1:8b", "messages": [{"role": "user", "content": "Hello"}], "stream": false }'

Embeddings

curl http://localhost:11434/v1/embeddings
-H "Content-Type: application/json"
-d '{ "model": "nomic-embed-text", "input": "The quick brown fox" }'

List models

curl http://localhost:11434/api/tags

undefined

curl http://localhost:11434/api/tags

undefined

Python Client

Python 客户端

python

undefined

python

undefined

pip install ollama

import ollama

Chat

response = ollama.chat( model="llama3.1:8b", messages=[{"role": "user", "content": "Explain Docker in 3 sentences"}], ) print(response["message"]["content"])

Streaming

for chunk in ollama.chat( model="llama3.1:8b", messages=[{"role": "user", "content": "Write a haiku about containers"}], stream=True, ): print(chunk["message"]["content"], end="", flush=True)

Embeddings

result = ollama.embed(model="nomic-embed-text", input="Hello world") print(f"Embedding dimensions: {len(result['embeddings'][0])}")

undefined

result = ollama.embed(model="nomic-embed-text", input="Hello world") print(f"Embedding dimensions: {len(result['embeddings'][0])}")

undefined

OpenAI SDK Compatibility

OpenAI SDK 兼容性

python

from openai import OpenAI

client = OpenAI(base_url="http://localhost:11434/v1", api_key="unused")

response = client.chat.completions.create(
    model="llama3.1:8b",
    messages=[{"role": "user", "content": "Hello"}],
)
print(response.choices[0].message.content)

python

from openai import OpenAI

client = OpenAI(base_url="http://localhost:11434/v1", api_key="unused")

response = client.chat.completions.create(
    model="llama3.1:8b",
    messages=[{"role": "user", "content": "Hello"}],
)
print(response.choices[0].message.content)

Custom Modelfiles

自定义Modelfiles

Create specialized models with custom system prompts and parameters:

dockerfile

undefined

通过自定义系统提示词和参数创建专用模型：

dockerfile

undefined

Modelfile.devops-assistant

FROM llama3.1:8b

SYSTEM """You are a DevOps expert assistant. You provide concise, production-ready advice about infrastructure, CI/CD, containers, and cloud services. Always include relevant commands and config examples."""

PARAMETER temperature 0.3 PARAMETER top_p 0.9 PARAMETER num_ctx 8192 PARAMETER repeat_penalty 1.1


```bash

FROM llama3.1:8b

SYSTEM """You are a DevOps expert assistant. You provide concise, production-ready advice about infrastructure, CI/CD, containers, and cloud services. Always include relevant commands and config examples."""

PARAMETER temperature 0.3 PARAMETER top_p 0.9 PARAMETER num_ctx 8192 PARAMETER repeat_penalty 1.1


```bash

Build and use custom model

ollama create devops-assistant -f Modelfile.devops-assistant ollama run devops-assistant "Set up a GitHub Actions workflow for Docker builds"

undefined

ollama create devops-assistant -f Modelfile.devops-assistant ollama run devops-assistant "Set up a GitHub Actions workflow for Docker builds"

undefined

GPU Configuration

GPU 配置

NVIDIA

bash

undefined

bash

undefined

Verify GPU access

nvidia-smi ollama run llama3.1:8b --verbose # Shows GPU layers loaded

Environment tuning

export OLLAMA_NUM_PARALLEL=4 # Concurrent requests export OLLAMA_MAX_LOADED_MODELS=2 # Models in VRAM export OLLAMA_FLASH_ATTENTION=1 # Faster attention export CUDA_VISIBLE_DEVICES=0,1 # Multi-GPU

undefined

export OLLAMA_NUM_PARALLEL=4 # Concurrent requests export OLLAMA_MAX_LOADED_MODELS=2 # Models in VRAM export OLLAMA_FLASH_ATTENTION=1 # Faster attention export CUDA_VISIBLE_DEVICES=0,1 # Multi-GPU

undefined

Apple Silicon

bash

undefined

bash

undefined

Metal acceleration is automatic on macOS

Verify with:

ollama run llama3.1:8b --verbose

Look for: "metal" in the output

Optimize for unified memory

export OLLAMA_NUM_PARALLEL=2 # Keep memory headroom export OLLAMA_MAX_LOADED_MODELS=1 # One model at a time on 16GB

undefined

export OLLAMA_NUM_PARALLEL=2 # Keep memory headroom export OLLAMA_MAX_LOADED_MODELS=1 # One model at a time on 16GB

undefined

Monitoring

监控

bash

undefined

bash

undefined

Check running models and memory usage

curl http://localhost:11434/api/ps

Prometheus metrics (if enabled)

curl http://localhost:11434/metrics

Quick health check script

#!/bin/bash response=$(curl -s -o /dev/null -w "%{http_code}" http://localhost:11434/api/tags) if [ "$response" = "200" ]; then echo "Ollama is healthy" curl -s http://localhost:11434/api/ps | python3 -m json.tool else echo "Ollama is down (HTTP $response)" exit 1 fi

undefined

#!/bin/bash response=$(curl -s -o /dev/null -w "%{http_code}" http://localhost:11434/api/tags) if [ "$response" = "200" ]; then echo "Ollama is healthy" curl -s http://localhost:11434/api/ps | python3 -m json.tool else echo "Ollama is down (HTTP $response)" exit 1 fi

undefined

Systemd Service

Systemd 服务

ini

undefined

ini

undefined

/etc/systemd/system/ollama.service

[Unit] Description=Ollama LLM Server After=network-online.target Wants=network-online.target

[Service] ExecStart=/usr/local/bin/ollama serve User=ollama Group=ollama Restart=always RestartSec=3 Environment="OLLAMA_HOST=0.0.0.0" Environment="OLLAMA_NUM_PARALLEL=4" Environment="OLLAMA_FLASH_ATTENTION=1" LimitNOFILE=65535

[Install] WantedBy=default.target


```bash
sudo useradd -r -s /bin/false -m -d /usr/share/ollama ollama
sudo systemctl daemon-reload
sudo systemctl enable --now ollama
sudo systemctl status ollama

[Unit] Description=Ollama LLM Server After=network-online.target Wants=network-online.target

[Service] ExecStart=/usr/local/bin/ollama serve User=ollama Group=ollama Restart=always RestartSec=3 Environment="OLLAMA_HOST=0.0.0.0" Environment="OLLAMA_NUM_PARALLEL=4" Environment="OLLAMA_FLASH_ATTENTION=1" LimitNOFILE=65535

[Install] WantedBy=default.target


```bash
sudo useradd -r -s /bin/false -m -d /usr/share/ollama ollama
sudo systemctl daemon-reload
sudo systemctl enable --now ollama
sudo systemctl status ollama

Security

安全建议

Bind to
```
127.0.0.1
```
in production (default), use reverse proxy for remote access
Set
```
WEBUI_AUTH=true
```
on Open WebUI
Use nginx with TLS for remote access:

nginx

server {
    listen 443 ssl;
    server_name llm.internal.example.com;
    ssl_certificate /etc/ssl/certs/llm.pem;
    ssl_certificate_key /etc/ssl/private/llm.key;

    location / {
        proxy_pass http://127.0.0.1:11434;
        proxy_set_header Host $host;
        proxy_buffering off;              # Required for streaming
        proxy_read_timeout 600s;          # Long model responses
        allow 10.0.0.0/8;
        deny all;
    }
}

生产环境中绑定到
```
127.0.0.1
```
（默认设置），如需远程访问请使用反向代理
在Open WebUI中设置
```
WEBUI_AUTH=true
```
使用带TLS的nginx进行远程访问：

nginx

server {
    listen 443 ssl;
    server_name llm.internal.example.com;
    ssl_certificate /etc/ssl/certs/llm.pem;
    ssl_certificate_key /etc/ssl/private/llm.key;

    location / {
        proxy_pass http://127.0.0.1:11434;
        proxy_set_header Host $host;
        proxy_buffering off;              # Required for streaming
        proxy_read_timeout 600s;          # Long model responses
        allow 10.0.0.0/8;
        deny all;
    }
}

Troubleshooting

故障排查

Issue	Solution
Model too slow	Use smaller quantization ( `q4_K_M` ), enable flash attention
Out of memory	Reduce `num_ctx` , use smaller model, set `OLLAMA_MAX_LOADED_MODELS=1`
GPU not detected	Check `nvidia-smi` , reinstall CUDA drivers, verify Docker GPU runtime
Connection refused	Check `OLLAMA_HOST` setting, verify firewall rules
Model download fails	Check disk space, retry with `ollama pull --insecure` for self-signed registries

问题	解决方案
模型运行过慢	使用更小的量化版本（如 `q4_K_M` ），启用flash attention
内存不足	减小 `num_ctx` 参数，使用更小的模型，设置 `OLLAMA_MAX_LOADED_MODELS=1`
GPU未被检测到	检查 `nvidia-smi` 输出，重新安装CUDA驱动，验证Docker GPU运行时配置
连接被拒绝	检查 `OLLAMA_HOST` 设置，验证防火墙规则
模型下载失败	检查磁盘空间，对于自签名镜像仓库，使用 `ollama pull --insecure` 重试

ollama-stack

Original

Translation

Ollama Stack

Ollama 本地LLM栈

When to Use This Skill

适用场景

Prerequisites

前置条件

Quick Start

快速开始

Install Ollama

Install Ollama

Start the server

Start the server

Pull and run a model

Pull and run a model

List available models

List available models

Pull specific quantization

Pull specific quantization

Model Selection Guide

模型选择指南

Docker Compose — Full Stack

Docker Compose — 完整部署栈

docker-compose.yml

docker-compose.yml

LiteLLM Proxy Config

LiteLLM 代理配置

litellm-config.yaml

litellm-config.yaml

API Usage

API 使用方法

Chat completion

Chat completion

Embeddings

Embeddings

List models

List models

Python Client

Python 客户端

pip install ollama

pip install ollama

Chat

Chat

Streaming

Streaming

Embeddings

Embeddings

OpenAI SDK Compatibility

OpenAI SDK 兼容性

Custom Modelfiles

自定义Modelfiles

Modelfile.devops-assistant

Modelfile.devops-assistant

Build and use custom model

Build and use custom model

GPU Configuration

GPU 配置

NVIDIA

NVIDIA

Verify GPU access

Verify GPU access

Environment tuning

Environment tuning

Apple Silicon

Apple Silicon

Metal acceleration is automatic on macOS

Metal acceleration is automatic on macOS

Verify with:

Verify with:

Look for: "metal" in the output

Look for: "metal" in the output

Optimize for unified memory

Optimize for unified memory

Monitoring

监控

Check running models and memory usage

Check running models and memory usage

Prometheus metrics (if enabled)