LLM使用监控仪表盘

基于Tokuin CLI追踪LLM API的成本、令牌、延迟，自动生成可为PM提供数据驱动洞察的管理员仪表盘。

本技能适用场景

获取LLM成本可见性：需要实时监控团队/个人API使用成本时
需要PM报告用仪表盘：需要输出每周AI使用人员、用量、使用方式的周报时
管理用户 adoption 率：需要追踪非活跃用户、提升AI普及率时
准备模型优化依据：需要基于数据做模型切换、成本削减决策时
为现有管理员仪表盘新增监控 tab：需要给现有Admin页面添加LLM监控模块时

前置条件

1. Tokuin CLI 安装确认

bash

# 检查是否已安装
which tokuin && tokuin --version || echo "未安装 — 请先执行Step 1"

2. 环境变量（仅实际调用API时需要）

bash

# 存储在.env文件中（绝对禁止直接写在代码里）
OPENAI_API_KEY=sk-...          # OpenAI
ANTHROPIC_API_KEY=sk-ant-...   # Anthropic
OPENROUTER_API_KEY=sk-or-...   # OpenRouter (支持400+模型)

# LLM监控配置
LLM_USER_ID=dev-alice           # 用户标识符
LLM_USER_ALIAS=Alice            # 显示名称
COST_THRESHOLD_USD=10.00        # 成本阈值（超出发送通知）
DASHBOARD_PORT=3000             # 仪表盘端口
MAX_COST_USD=5.00               # 单次运行最大成本
SLACK_WEBHOOK_URL=https://...   # 通知用（可选）

3. 项目技术栈要求

Option A（推荐）: Next.js 15+ + React 18 + TypeScript
Option B（轻量版）: Python 3.8+ + HTML/JavaScript（依赖最少）

使用指南

Step 0: 安全检查（必须最先执行）

⚠️ 执行技能前务必运行本脚本，存在FAIL项会自动中断执行。

bash

cat > safety-guard.sh << 'SAFETY_EOF'
#!/usr/bin/env bash
# safety-guard.sh — LLM监控仪表盘运行前安全检查
set -euo pipefail

RED='\033[0;31m'; YELLOW='\033[1;33m'; GREEN='\033[0;32m'; NC='\033[0m'
ALLOW_LIVE="${1:-}"; PASS=0; WARN=0; FAIL=0

log_pass() { echo -e "${GREEN}✅ PASS${NC} $1"; ((PASS++)); }
log_warn() { echo -e "${YELLOW}⚠️  WARN${NC} $1"; ((WARN++)); }
log_fail() { echo -e "${RED}❌ FAIL${NC} $1"; ((FAIL++)); }

echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
echo "🛡  LLM Monitoring Dashboard — Safety Guard v1.0"
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"

# ── 1. Tokuin CLI 安装检查 ────────────────────────────────
if command -v tokuin &>/dev/null; then
  log_pass "Tokuin CLI 已安装: $(tokuin --version 2>&1 | head -1)"
else
  log_fail "未安装Tokuin → 执行以下命令安装后重试:"
  echo "  curl -fsSL https://raw.githubusercontent.com/nooscraft/tokuin/main/install.sh | bash"
fi

# ── 2. 检测API密钥硬编码 ────────────────────────────────
HARDCODED=$(grep -rE "(sk-[a-zA-Z0-9]{20,}|sk-ant-[a-zA-Z0-9]{20,}|sk-or-[a-zA-Z0-9]{20,})" \
  . --include="*.ts" --include="*.tsx" --include="*.js" --include="*.jsx" \
  --include="*.html" --include="*.sh" --include="*.py" --include="*.json" \
  --exclude-dir=node_modules --exclude-dir=.git 2>/dev/null \
  | grep -v "\.env" | grep -v "example" | wc -l || echo 0)
if [ "$HARDCODED" -eq 0 ]; then
  log_pass "未检测到API密钥硬编码"
else
  log_fail "⚠️  检测到${HARDCODED}处API密钥硬编码! → 请立即迁移到环境变量(.env)中"
  grep -rE "(sk-[a-zA-Z0-9]{20,})" . \
    --include="*.ts" --include="*.js" --include="*.html" \
    --exclude-dir=node_modules 2>/dev/null | head -5 || true
fi

# ── 3. 检查.env是否已加入.gitignore ────────────────────────
if [ -f .env ]; then
  if [ -f .gitignore ] && grep -q "\.env" .gitignore; then
    log_pass ".env已加入.gitignore"
  else
    log_fail ".env存在但未加入.gitignore! → 执行 echo '.env' >> .gitignore"
  fi
else
  log_warn "无.env文件 — 实际调用API时需要创建"
fi

# ── 4. 确认是否为真实API调用模式 ────────────────────────────
if [ "$ALLOW_LIVE" = "--allow-live" ]; then
  log_warn "真实API调用模式已开启! 会产生实际费用。"
  log_warn "最大成本阈值: \$${MAX_COST_USD:-5.00} (可通过MAX_COST_USD环境变量调整)"
  read -p "  确认允许真实API调用吗? [y/N] " -r
  echo
  [[ $REPLY =~ ^[Yy]$ ]] || { echo "已取消。请使用dry-run模式重新运行。"; exit 1; }
else
  log_pass "dry-run模式（默认） — 不会产生API费用"
fi

# ── 5. 端口冲突检查 ─────────────────────────────────────
PORT="${DASHBOARD_PORT:-3000}"
if lsof -i ":${PORT}" &>/dev/null 2>&1; then
  ALT_PORT=$((PORT + 1))
  log_warn "端口${PORT}已被占用 → 将使用${ALT_PORT}端口: export DASHBOARD_PORT=${ALT_PORT}"
else
  log_pass "端口${PORT}可用"
fi

# ── 6. 初始化data/目录 ──────────────────────────────
mkdir -p ./data
if [ -f ./data/metrics.jsonl ]; then
  BYTES=$(wc -c < ./data/metrics.jsonl || echo 0)
  if [ "$BYTES" -gt 10485760 ]; then
    log_warn "metrics.jsonl已超过10MB (${BYTES}B) → 建议配置滚动存储策略"
    echo "  cp data/metrics.jsonl data/metrics-$(date +%Y%m%d).jsonl.bak && > data/metrics.jsonl"
  else
    log_pass "data/目录已就绪 (metrics.jsonl: ${BYTES}B)"
  fi
else
  log_pass "data/目录已就绪（新建）"
fi

# ── 结果汇总 ─────────────────────────────────────────────
echo ""
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
echo -e "结果: ${GREEN}PASS $PASS${NC} / ${YELLOW}WARN $WARN${NC} / ${RED}FAIL $FAIL${NC}"
if [ "$FAIL" -gt 0 ]; then
  echo -e "${RED}❌ 安全检查失败。请解决上述FAIL项后重试。${NC}"
  exit 1
else
  echo -e "${GREEN}✅ 安全检查通过。继续执行技能。${NC}"
  exit 0
fi
SAFETY_EOF
chmod +x safety-guard.sh

# 执行（存在FAIL项会立即中断）
bash safety-guard.sh

Step 1: 安装Tokuin CLI并验证dry-run

bash

# 1-1. 安装（macOS / Linux）
curl -fsSL https://raw.githubusercontent.com/nooscraft/tokuin/main/install.sh | bash

# Windows PowerShell:
# irm https://raw.githubusercontent.com/nooscraft/tokuin/main/install.ps1 | iex

# 1-2. 确认安装成功
tokuin --version
which tokuin   # 预期输出: /usr/local/bin/tokuin 或 ~/.local/bin/tokuin

# 1-3. 基础令牌计数测试
echo "Hello, world!" | tokuin --model gpt-4

# 1-4. dry-run成本估算（不需要API密钥 ✅）
echo "Analyze user behavior patterns from the following data" | \
  tokuin load-test \
  --model gpt-4 \
  --runs 50 \
  --concurrency 5 \
  --dry-run \
  --estimate-cost \
  --output-format json | python3 -m json.tool

# 预期输出结构:
# {
#   "total_requests": 50,
#   "successful": 50,
#   "failed": 0,
#   "latency_ms": { "average": ..., "p50": ..., "p95": ... },
#   "cost": { "input_tokens": ..., "output_tokens": ..., "total_cost": ... }
# }

# 1-5. 多模型对比（dry-run）
echo "Translate this to Korean" | tokuin --compare gpt-4 gpt-3.5-turbo claude-3-haiku --price

# 1-6. 确认Prometheus格式输出
echo "Benchmark" | tokuin load-test --model gpt-4 --runs 10 --dry-run --output-format prometheus
# 预期输出: 带"# HELP", "# TYPE", "tokuin_"前缀的指标

Step 2: 包含用户上下文的数据采集管道

bash

# 2-1. 创建提示词自动分类模块
cat > categorize_prompt.py << 'PYEOF'
#!/usr/bin/env python3
"""基于关键词自动分类提示词"""
import hashlib

CATEGORIES = {
    "编码":   ["code", "function", "class", "implement", "debug", "fix", "refactor", "코드", "구현", "함수"],
    "分析":   ["analyze", "compare", "evaluate", "assess", "분석", "비교", "평가", "검토"],
    "翻译":   ["translate", "translation", "번역", "영어로", "한국어로"],
    "摘要":   ["summarize", "summary", "tldr", "brief", "요약", "정리"],
    "写作":   ["write", "draft", "create", "generate", "작성", "생성", "만들어"],
    "提问":   ["what is", "how to", "explain", "why", "무엇", "어떻게", "설명", "왜"],
    "数据": ["data", "table", "csv", "json", "sql", "데이터", "테이블", "쿼리"],
}

def categorize(prompt: str) -> str:
    p = prompt.lower()
    for cat, keywords in CATEGORIES.items():
        if any(k in p for k in keywords):
            return cat
    return "其他"

def hash_prompt(prompt: str) -> str:
    """取SHA-256前16位（存储哈希而非原文，保护隐私）"""
    return hashlib.sha256(prompt.encode()).hexdigest()[:16]

def truncate_preview(prompt: str, limit: int = 100) -> str:
    return prompt[:limit] + ("…" if len(prompt) > limit else "")

if __name__ == "__main__":
    import sys
    prompt = sys.argv[1] if len(sys.argv) > 1 else ""
    print(categorize(prompt))
PYEOF

# 2-2. 创建包含用户上下文的指标采集脚本
cat > collect-metrics.sh << 'COLLECT_EOF'
#!/usr/bin/env bash
# collect-metrics.sh — 运行Tokuin + 存储用户上下文（默认dry-run）
set -euo pipefail

# 用户信息
USER_ID="${LLM_USER_ID:-$(whoami)}"
USER_ALIAS="${LLM_USER_ALIAS:-$USER_ID}"
SESSION_ID="${LLM_SESSION_ID:-$(date +%Y%m%d-%H%M%S)-$$}"
PROMPT="${1:-Benchmark prompt}"
MODEL="${MODEL:-gpt-4}"
PROVIDER="${PROVIDER:-openai}"
RUNS="${RUNS:-50}"
CONCURRENCY="${CONCURRENCY:-5}"
TAGS="${LLM_TAGS:-[]}"

TIMESTAMP=$(date -u +"%Y-%m-%dT%H:%M:%SZ")
CATEGORY=$(python3 categorize_prompt.py "$PROMPT" 2>/dev/null || echo "其他")
PROMPT_HASH=$(echo -n "$PROMPT" | sha256sum | cut -c1-16 2>/dev/null || echo "unknown")
PROMPT_LEN=${#PROMPT}

# 运行Tokuin（默认dry-run）
RESULT=$(echo "$PROMPT" | tokuin load-test \
  --model "$MODEL" \
  --provider "$PROVIDER" \
  --runs "$RUNS" \
  --concurrency "$CONCURRENCY" \
  --output-format json \
  ${ALLOW_LIVE:+""} ${ALLOW_LIVE:-"--dry-run --estimate-cost"} 2>/dev/null)

# 包含用户上下文保存为JSONL
python3 - << PYEOF
import json, sys

result = json.loads('''${RESULT}''')
latency = result.get("latency_ms", {})
cost = result.get("cost", {})

record = {
    "id": "${PROMPT_HASH}-${SESSION_ID}",
    "timestamp": "${TIMESTAMP}",
    "model": "${MODEL}",
    "provider": "${PROVIDER}",
    "user_id": "${USER_ID}",
    "user_alias": "${USER_ALIAS}",
    "session_id": "${SESSION_ID}",
    "prompt_hash": "${PROMPT_HASH}",
    "prompt_category": "${CATEGORY}",
    "prompt_length": ${PROMPT_LEN},
    "tags": json.loads('${TAGS}'),
    "is_dry_run": True,
    "total_requests": result.get("total_requests", 0),
    "successful": result.get("successful", 0),
    "failed": result.get("failed", 0),
    "input_tokens": cost.get("input_tokens", 0),
    "output_tokens": cost.get("output_tokens", 0),
    "cost_usd": cost.get("total_cost", 0),
    "latency_avg_ms": latency.get("average", 0),
    "latency_p50_ms": latency.get("p50", 0),
    "latency_p95_ms": latency.get("p95", 0),
    "status_code": 200 if result.get("successful", 0) > 0 else 500,
}

with open("./data/metrics.jsonl", "a") as f:
    f.write(json.dumps(record, ensure_ascii=False) + "\n")

print(f"✅ 已保存: [{record['user_alias']}] {record['prompt_category']} | ${record['cost_usd']:.4f} | {record['latency_avg_ms']:.0f}ms")
PYEOF
COLLECT_EOF
chmod +x collect-metrics.sh

# 2-3. 配置cron（每5分钟自动采集）
(crontab -l 2>/dev/null; echo "*/5 * * * * cd $(pwd) && bash collect-metrics.sh 'Scheduled benchmark' >> ./data/collect.log 2>&1") | crontab -
echo "✅ cron定时任务注册完成（每5分钟运行一次）"

# 2-4. 首次采集测试（dry-run）
bash collect-metrics.sh "Analyze user behavior patterns"
cat ./data/metrics.jsonl | python3 -m json.tool | head -30

Step 3: 创建路由结构和仪表盘框架

Option A — Next.js（推荐）

bash

# 3-1. 初始化Next.js项目（添加到现有项目可跳过本步）
npx create-next-app@latest llm-dashboard \
  --typescript \
  --tailwind \
  --app \
  --no-src-dir
cd llm-dashboard

# 3-2. 安装依赖
npm install recharts better-sqlite3 @types/better-sqlite3

# 3-3. 配置设计token（统一设计风格）
cat > app/globals.css << 'CSS_EOF'
:root {
  /* 背景层级 */
  --bg-base:     #0f1117;
  --bg-surface:  #1a1d27;
  --bg-elevated: #21253a;
  --border:      rgba(255, 255, 255, 0.06);

  /* 文本层级 */
  --text-primary:   #f1f5f9;
  --text-secondary: #94a3b8;
  --text-muted:     #475569;

  /* 三级信号灯系统（所有组件统一使用） */
  --color-ok:      #22c55e;   /* 正常 — Green 500 */
  --color-warn:    #f59e0b;   /* 警告 — Amber 500 */
  --color-danger:  #ef4444;   /* 危险 — Red 500   */
  --color-neutral: #60a5fa;   /* 中性 — Blue 400  */

  /* 数据系列颜色（兼容色盲的调色板） */
  --series-1: #818cf8;  /* Indigo  — 系统/GPT-4    */
  --series-2: #38bdf8;  /* Sky     — 用户/Claude     */
  --series-3: #34d399;  /* Emerald — 助理/Gemini*/
  --series-4: #fb923c;  /* Orange  — 第4个系列    */

  /* 成本专属颜色 */
  --cost-input:  #a78bfa;
  --cost-output: #f472b6;

  /* 排名颜色 */
  --rank-gold:     #fbbf24;
  --rank-silver:   #94a3b8;
  --rank-bronze:   #b45309;
  --rank-inactive: #374151;

  /* 排版 */
  --font-mono: 'JetBrains Mono', 'Fira Code', monospace;
  --font-ui:   'Geist', 'Plus Jakarta Sans', system-ui, sans-serif;
}

body {
  background: var(--bg-base);
  color: var(--text-primary);
  font-family: var(--font-ui);
}

/* 数字显示：对齐稳定 */
.metric-value {
  font-family: var(--font-mono);
  font-variant-numeric: tabular-nums;
  font-feature-settings: 'tnum';
}

/* KPI卡片 accent-bar */
.status-ok     { border-left-color: var(--color-ok); }
.status-warn   { border-left-color: var(--color-warn); }
.status-danger { border-left-color: var(--color-danger); }
CSS_EOF

# 3-4. 创建路由结构
mkdir -p app/admin/llm-monitoring
mkdir -p app/admin/llm-monitoring/users
mkdir -p "app/admin/llm-monitoring/users/[userId]"
mkdir -p "app/admin/llm-monitoring/runs/[runId]"
mkdir -p components/llm-monitoring
mkdir -p lib/llm-monitoring

# 3-5. 初始化SQLite DB
cat > lib/llm-monitoring/db.ts << 'TS_EOF'
import Database from 'better-sqlite3'
import path from 'path'

const DB_PATH = path.join(process.cwd(), 'data', 'monitoring.db')

const db = new Database(DB_PATH)

db.exec(`
  CREATE TABLE IF NOT EXISTS runs (
    id              TEXT PRIMARY KEY,
    timestamp       DATETIME NOT NULL DEFAULT (datetime('now')),
    model           TEXT NOT NULL,
    provider        TEXT NOT NULL,
    user_id         TEXT DEFAULT 'anonymous',
    user_alias      TEXT DEFAULT 'anonymous',
    session_id      TEXT,
    prompt_hash     TEXT,
    prompt_category TEXT DEFAULT '其他',
    prompt_length   INTEGER DEFAULT 0,
    tags            TEXT DEFAULT '[]',
    is_dry_run      INTEGER DEFAULT 1,
    total_requests  INTEGER DEFAULT 0,
    successful      INTEGER DEFAULT 0,
    failed          INTEGER DEFAULT 0,
    input_tokens    INTEGER DEFAULT 0,
    output_tokens   INTEGER DEFAULT 0,
    cost_usd        REAL DEFAULT 0,
    latency_avg_ms  REAL DEFAULT 0,
    latency_p50_ms  REAL DEFAULT 0,
    latency_p95_ms  REAL DEFAULT 0,
    status_code     INTEGER DEFAULT 200
  );

  CREATE TABLE IF NOT EXISTS user_profiles (
    user_id    TEXT PRIMARY KEY,
    user_alias TEXT NOT NULL,
    team       TEXT DEFAULT '',
    role       TEXT DEFAULT 'user',
    created_at DATETIME DEFAULT (datetime('now')),
    last_seen  DATETIME,
    notes      TEXT DEFAULT ''
  );

  CREATE INDEX IF NOT EXISTS idx_runs_timestamp  ON runs(timestamp DESC);
  CREATE INDEX IF NOT EXISTS idx_runs_user_id    ON runs(user_id);
  CREATE INDEX IF NOT EXISTS idx_runs_model      ON runs(model);

  CREATE VIEW IF NOT EXISTS user_stats AS
  SELECT
    user_id,
    user_alias,
    COUNT(*)                          AS total_runs,
    SUM(input_tokens + output_tokens) AS total_tokens,
    ROUND(SUM(cost_usd), 4)           AS total_cost,
    ROUND(AVG(latency_avg_ms), 1)     AS avg_latency,
    ROUND(AVG(CAST(successful AS REAL) / NULLIF(total_requests, 0) * 100), 1) AS success_rate,
    COUNT(DISTINCT model)             AS models_used,
    MAX(timestamp)                    AS last_seen
  FROM runs
  GROUP BY user_id;
`)

export default db
TS_EOF

Option B — 轻量HTML（最小依赖）

bash

# 无现有项目或需要快速原型时使用
mkdir -p llm-monitoring/data

cat > llm-monitoring/index.html << 'HTML_EOF'
<!DOCTYPE html>
<html lang="zh-CN">
<head>
  <meta charset="UTF-8">
  <meta name="viewport" content="width=device-width, initial-scale=1.0">
  <title>🧮 LLM使用监控</title>
  <script src="https://cdn.jsdelivr.net/npm/chart.js@4/dist/chart.umd.min.js"></script>
  <link rel="preconnect" href="https://fonts.googleapis.com">
  <link href="https://fonts.googleapis.com/css2?family=JetBrains+Mono:wght@400;600&display=swap" rel="stylesheet">
  <style>
    /* 设计token */
    :root {
      --bg-base: #0f1117; --bg-surface: #1a1d27; --bg-elevated: #21253a;
      --text-primary: #f1f5f9; --text-secondary: #94a3b8; --text-muted: #475569;
      --color-ok: #22c55e; --color-warn: #f59e0b; --color-danger: #ef4444;
      --series-1: #818cf8; --series-2: #38bdf8; --series-3: #34d399; --series-4: #fb923c;
      --rank-gold: #fbbf24; --rank-silver: #94a3b8; --rank-bronze: #b45309;
      --font-mono: 'JetBrains Mono', monospace;
    }
    * { box-sizing: border-box; margin: 0; padding: 0; }
    body { background: var(--bg-base); color: var(--text-primary); font-family: system-ui, sans-serif; padding: 24px; }
    header { display: flex; justify-content: space-between; align-items: center; margin-bottom: 32px; }
    header h1 { font-size: 1.5rem; font-weight: 700; color: #60a5fa; }
    .kpi-grid { display: grid; grid-template-columns: repeat(4, 1fr); gap: 16px; margin-bottom: 24px; }
    @media (max-width: 768px) { .kpi-grid { grid-template-columns: repeat(2, 1fr); } }
    @media (max-width: 480px) { .kpi-grid { grid-template-columns: 1fr; } }
    .kpi-card {
      background: var(--bg-surface);
      border: 1px solid rgba(255,255,255,0.06);
      border-left: 3px solid var(--color-neutral, #60a5fa);
      border-radius: 12px;
      padding: 20px;
    }
    .kpi-card.ok     { border-left-color: var(--color-ok); }
    .kpi-card.warn   { border-left-color: var(--color-warn); }
    .kpi-card.danger { border-left-color: var(--color-danger); }
    .kpi-label { font-size: 0.625rem; text-transform: uppercase; letter-spacing: 0.1em; color: var(--text-muted); margin-bottom: 8px; }
    .kpi-value { font-family: var(--font-mono); font-size: 2rem; font-weight: 700; font-variant-numeric: tabular-nums; }
    .kpi-sub   { font-size: 0.75rem; color: var(--text-secondary); margin-top: 4px; }
    .chart-row { display: grid; grid-template-columns: 2fr 1fr; gap: 16px; margin-bottom: 24px; }
    @media (max-width: 900px) { .chart-row { grid-template-columns: 1fr; } }
    .chart-card { background: var(--bg-surface); border: 1px solid rgba(255,255,255,0.06); border-radius: 12px; padding: 20px; }
    .chart-card h3 { font-size: 0.75rem; color: var(--text-secondary); margin-bottom: 16px; text-transform: uppercase; letter-spacing: 0.05em; }
    .ranking-table { width: 100%; border-collapse: collapse; }
    .ranking-table th { font-size: 0.625rem; text-transform: uppercase; color: var(--text-muted); padding: 8px 12px; text-align: left; border-bottom: 1px solid rgba(255,255,255,0.06); }
    .ranking-table td { padding: 12px; border-bottom: 1px solid rgba(255,255,255,0.04); font-family: var(--font-mono); font-size: 0.875rem; }
    .ranking-table tr:hover td { background: var(--bg-elevated); }
    .user-link { color: #60a5fa; text-decoration: none; cursor: pointer; }
    .user-link:hover { text-decoration: underline; }
    .badge { display: inline-block; padding: 2px 8px; border-radius: 4px; font-size: 0.7rem; }
    .badge-ok     { background: rgba(34,197,94,0.1);  color: var(--color-ok); }
    .badge-warn   { background: rgba(245,158,11,0.1); color: var(--color-warn); }
    .badge-danger { background: rgba(239,68,68,0.1);  color: var(--color-danger); }
    .rank-1 { color: var(--rank-gold); }
    .rank-2 { color: var(--rank-silver); }
    .rank-3 { color: var(--rank-bronze); }
    .insight-box { background: rgba(96,165,250,0.05); border: 1px solid rgba(96,165,250,0.15); border-radius: 8px; padding: 16px; margin-top: 8px; }
    .insight-box h4 { font-size: 0.75rem; color: #60a5fa; margin-bottom: 8px; }
    .insight-box ul { font-size: 0.8rem; color: var(--text-secondary); padding-left: 16px; }
    .insight-box ul li { margin-bottom: 4px; }
    .section-title { font-size: 1rem; font-weight: 600; margin: 24px 0 12px; }
    #user-detail { display: none; background: var(--bg-surface); border: 1px solid rgba(255,255,255,0.06); border-radius: 12px; padding: 24px; margin-top: 16px; }
    .back-btn { background: none; border: 1px solid rgba(255,255,255,0.1); color: var(--text-secondary); padding: 6px 12px; border-radius: 6px; cursor: pointer; font-size: 0.8rem; margin-bottom: 16px; }
    .back-btn:hover { background: var(--bg-elevated); }
  </style>
</head>
<body>
  <header>
    <div>
      <h1>🧮 LLM使用监控</h1>
      <p style="font-size:0.75rem;color:#475569;margin-top:4px;">Powered by Tokuin CLI</p>
    </div>
    <div style="display:flex;gap:8px;align-items:center;">
      <span id="last-updated" style="font-size:0.75rem;color:#475569;"></span>
      <button onclick="loadData()" style="background:rgba(96,165,250,0.1);border:1px solid rgba(96,165,250,0.2);color:#60a5fa;padding:6px 14px;border-radius:6px;cursor:pointer;font-size:0.8rem;">↻ 刷新</button>
    </div>
  </header>

  <!-- 主仪表盘 -->
  <div id="main-dashboard">
    <!-- 4种KPI卡片 -->
    <div class="kpi-grid">
      <div class="kpi-card" id="kpi-requests">
        <div class="kpi-label">总请求数</div>
        <div class="kpi-value metric-value" id="val-requests">-</div>
        <div class="kpi-sub" id="sub-requests">数据加载中...</div>
      </div>
      <div class="kpi-card" id="kpi-success">
        <div class="kpi-label">成功率</div>
        <div class="kpi-value metric-value" id="val-success">-</div>
        <div class="kpi-sub" id="sub-success">-</div>
      </div>
      <div class="kpi-card" id="kpi-latency">
        <div class="kpi-label">p95 延迟</div>
        <div class="kpi-value metric-value" id="val-latency">-</div>
        <div class="kpi-sub" id="sub-latency">-</div>
      </div>
      <div class="kpi-card" id="kpi-cost">
        <div class="kpi-label">总成本</div>
        <div class="kpi-value metric-value" id="val-cost">-</div>
        <div class="kpi-sub" id="sub-cost">-</div>
      </div>
    </div>

    <!-- 图表行 -->
    <div class="chart-row">
      <div class="chart-card">
        <h3>分时段成本趋势</h3>
        <canvas id="trend-chart" height="160"></canvas>
      </div>
      <div class="chart-card">
        <h3>分类分布</h3>
        <canvas id="category-chart" height="160"></canvas>
      </div>
    </div>

    <!-- 用户排名 -->
    <h2 class="section-title">🏆 用户排名</h2>
    <div class="chart-card" style="margin-bottom:24px;">
      <div style="display:flex;justify-content:space-between;align-items:center;margin-bottom:12px;">
        <h3 style="margin-bottom:0;">按成本排名</h3>
        <input id="user-search" type="text" placeholder="🔍 搜索用户..." 
          style="background:var(--bg-elevated);border:1px solid rgba(255,255,255,0.08);color:var(--text-primary);padding:6px 12px;border-radius:6px;font-size:0.8rem;width:200px;"
          oninput="filterRanking(this.value)">
      </div>
      <table class="ranking-table" id="ranking-table">
        <thead>
          <tr>
            <th>排名</th>
            <th>用户</th>
            <th>成本</th>
            <th>请求数</th>
            <th>偏好模型</th>
            <th>成功率</th>
            <th>最后活动</th>
          </tr>
        </thead>
        <tbody id="ranking-body">
          <tr><td colspan="7" style="text-align:center;color:#475569;padding:24px;">数据加载中...</td></tr>
        </tbody>
      </table>
    </div>

    <!-- 非活跃用户追踪 -->
    <h2 class="section-title">💤 非活跃用户现状</h2>
    <div class="chart-card" style="margin-bottom:24px;">
      <table class="ranking-table" id="inactive-table">
        <thead>
          <tr><th>用户</th><th>未使用时长</th><th>最后活动</th><th>状态</th></tr>
        </thead>
        <tbody id="inactive-body">
          <tr><td colspan="4" style="text-align:center;color:#475569;padding:24px;">无追踪数据</td></tr>
        </tbody>
      </table>
    </div>

    <!-- PM洞察 -->
    <h2 class="section-title">📊 PM自动洞察</h2>
    <div id="pm-insights">
      <div class="insight-box">
        <h4>💡 自动分析中...</h4>
      </div>
    </div>
  </div>

  <!-- 用户个人详情页（点击链接显示） -->
  <div id="user-detail">
    <button class="back-btn" onclick="showMain()">← 返回仪表盘</button>
    <div id="user-detail-content"></div>
  </div>

  <script>
  let allData = [];
  let allUsers = {};

  async function loadData() {
    try {
      const res = await fetch('./data/metrics.jsonl');
      const text = await res.text();
      allData = text.trim().split('\n').filter(Boolean).map(l => JSON.parse(l));
      document.getElementById('last-updated').textContent = '最后更新: ' + new Date().toLocaleTimeString('zh-CN');
      renderDashboard();
    } catch(e) {
      // 无JSONL文件则显示样例数据
      allData = generateSampleData();
      renderDashboard();
    }
  }

  function generateSampleData() {
    const users = ['dev-alice', 'team-backend', 'analyst-bob', 'pm-charlie'];
    const models = ['gpt-4', 'claude-3-sonnet', 'gemini-pro'];
    const categories = ['编码', '分析', '翻译', '摘要', '写作'];
    const data = [];
    for (let i = 0; i < 50; i++) {
      const user = users[Math.floor(Math.random() * users.length)];
      const daysAgo = Math.floor(Math.random() * 30);
      const ts = new Date(Date.now() - daysAgo * 86400000 - Math.random() * 86400000);
      data.push({
        id: 'sample-' + i,
        timestamp: ts.toISOString(),
        model: models[Math.floor(Math.random() * models.length)],
        provider: 'openai',
        user_id: user,
        user_alias: user,
        prompt_category: categories[Math.floor(Math.random() * categories.length)],
        input_tokens: Math.floor(Math.random() * 2000) + 100,
        output_tokens: Math.floor(Math.random() * 1000) + 50,
        cost_usd: (Math.random() * 0.05).toFixed(4) * 1,
        latency_avg_ms: Math.floor(Math.random() * 1500) + 200,
        latency_p95_ms: Math.floor(Math.random() * 2500) + 500,
        successful: 1,
        total_requests: 1,
        is_dry_run: true,
        status_code: Math.random() > 0.05 ? 200 : 429,
      });
    }
    return data;
  }

  function renderDashboard() {
    if (!allData.length) return;

    // 计算KPI
    const totalReqs  = allData.reduce((s, r) => s + (r.total_requests || 1), 0);
    const totalSucc  = allData.filter(r => r.status_code === 200).length;
    const successRate = ((totalSucc / allData.length) * 100).toFixed(1);
    const avgLatency = (allData.reduce((s, r) => s + (r.latency_avg_ms || 0), 0) / allData.length).toFixed(0);
    const p95Latency = (allData.reduce((s, r) => s + (r.latency_p95_ms || 0), 0) / allData.length).toFixed(0);
    const totalCost  = allData.reduce((s, r) => s + (r.cost_usd || 0), 0).toFixed(4);

    // 更新KPI卡片
    document.getElementById('val-requests').textContent = totalReqs.toLocaleString();
    document.getElementById('sub-requests').textContent = allData.length + '条运行记录';

    document.getElementById('val-success').textContent = successRate + '%';
    document.getElementById('sub-success').textContent = '失败 ' + (allData.length - totalSucc) + '次';
    const kpiSuccess = document.getElementById('kpi-success');
    kpiSuccess.className = 'kpi-card ' + (successRate >= 95 ? 'ok' : successRate >= 90 ? 'warn' : 'danger');

    document.getElementById('val-latency').textContent = p95Latency + 'ms';
    document.getElementById('sub-latency').textContent = '平均 ' + avgLatency + 'ms';
    const kpiLatency = document.getElementById('kpi-latency');
    kpiLatency.className = 'kpi-card ' + (p95Latency < 1000 ? 'ok' : p95Latency < 2000 ? 'warn' : 'danger');

    document.getElementById('val-cost').textContent = '$' + totalCost;
    document.getElementById('sub-cost').textContent = 'dry-run估算值';

    // 趋势图
    renderTrendChart();
    // 分类分布
    renderCategoryChart();
    // 用户排名
    renderRanking();
    // 非活跃用户
    renderInactive();
    // PM洞察
    renderInsights(successRate, p95Latency, totalCost);
  }

  function renderTrendChart() {
    const ctx = document.getElementById('trend-chart').getContext('2d');
    const byDate = {};
    allData.forEach(r => {
      const d = r.timestamp.substring(0, 10);
      byDate[d] = (byDate[d] || 0) + (r.cost_usd || 0);
    });
    const labels = Object.keys(byDate).sort().slice(-14);
    const values = labels.map(d => byDate[d].toFixed(4));
    if (window._trendChart) window._trendChart.destroy();
    window._trendChart = new Chart(ctx, {
      type: 'line',
      data: {
        labels,
        datasets: [{
          label: '日成本 ($)',
          data: values,
          borderColor: '#818cf8',
          backgroundColor: 'rgba(129,140,248,0.1)',
          fill: true,
          tension: 0.4,
          pointRadius: 4,
          pointBackgroundColor: '#818cf8',
        }]
      },
      options: {
        plugins: { legend: { labels: { color: '#94a3b8' } } },
        scales: {
          x: { ticks: { color: '#475569' }, grid: { color: 'rgba(255,255,255,0.04)' } },
          y: { ticks: { color: '#475569' }, grid: { color: 'rgba(255,255,255,0.04)' } }
        }
      }
    });
  }

  function renderCategoryChart() {
    const ctx = document.getElementById('category-chart').getContext('2d');
    const cats = {};
    allData.forEach(r => { cats[r.prompt_category || '其他'] = (cats[r.prompt_category || '其他'] || 0) + 1; });
    const colors = ['#818cf8','#38bdf8','#34d399','#fb923c','#f472b6','#94a3b8'];
    if (window._catChart) window._catChart.destroy();
    window._catChart = new Chart(ctx, {
      type: 'doughnut',
      data: {
        labels: Object.keys(cats),
        datasets: [{ data: Object.values(cats), backgroundColor: colors, borderWidth: 0 }]
      },
      options: {
        plugins: { legend: { position: 'right', labels: { color: '#94a3b8', font: { size: 11 } } } },
        cutout: '65%'
      }
    });
  }

  function renderRanking(filter = '') {
    const userMap = {};
    allData.forEach(r => {
      const uid = r.user_id || 'anonymous';
      if (!userMap[uid]) userMap[uid] = { alias: r.user_alias || uid, cost: 0, runs: 0, models: {}, success: 0, last: r.timestamp };
      userMap[uid].cost  += r.cost_usd || 0;
      userMap[uid].runs  += 1;
      userMap[uid].models[r.model] = (userMap[uid].models[r.model] || 0) + 1;
      if (r.status_code === 200) userMap[uid].success++;
      if (r.timestamp > userMap[uid].last) userMap[uid].last = r.timestamp;
    });
    allUsers = userMap;
    const sorted = Object.entries(userMap)
      .filter(([uid, u]) => !filter || u.alias.toLowerCase().includes(filter.toLowerCase()))
      .sort((a, b) => b[1].cost - a[1].cost);
    const tbody = document.getElementById('ranking-body');
    if (!sorted.length) { tbody.innerHTML = '<tr><td colspan="7" style="text-align:center;color:#475569;padding:16px;">无搜索结果</td></tr>'; return; }
    const rankEmoji = ['🥇','🥈','🥉'];
    tbody.innerHTML = sorted.map(([uid, u], i) => {
      const topModel = Object.entries(u.models).sort((a,b) => b[1]-a[1])[0]?.[0] || '-';
      const sr = ((u.success / u.runs) * 100).toFixed(1);
      const srClass = sr >= 95 ? 'badge-ok' : sr >= 90 ? 'badge-warn' : 'badge-danger';
      const lastAgo = Math.floor((Date.now() - new Date(u.last)) / 86400000);
      const rankClass = i === 0 ? 'rank-1' : i === 1 ? 'rank-2' : i === 2 ? 'rank-3' : '';
      return `<tr>
        <td class="${rankClass}">${rankEmoji[i] || (i+1)}</td>
        <td><a class="user-link" onclick="showUserDetail('${uid}')">${u.alias}</a></td>
        <td class="metric-value">$${u.cost.toFixed(4)}</td>
        <td class="metric-value">${u.runs.toLocaleString()}</td>
        <td><span style="font-size:0.75rem;color:#94a3b8;">${topModel}</span></td>
        <td><span class="badge ${srClass}">${sr}%</span></td>
        <td style="color:#475569;font-size:0.75rem;">${lastAgo === 0 ? '今天' : lastAgo + '天前'}</td>
      </tr>`;
    }).join('');
  }

  function filterRanking(val) { renderRanking(val); }

  function renderInactive() {
    const sevenDaysAgo = new Date(Date.now() - 7 * 86400000);
    const activeUsers = new Set(
      allData.filter(r => new Date(r.timestamp) > sevenDaysAgo).map(r => r.user_id)
    );
    const lastSeen = {};
    allData.forEach(r => {
      if (!lastSeen[r.user_id] || r.timestamp > lastSeen[r.user_id].ts) {
        lastSeen[r.user_id] = { ts: r.timestamp, alias: r.user_alias || r.user_id };
      }
    });
    const inactive = Object.entries(lastSeen).filter(([uid]) => !activeUsers.has(uid));
    const tbody = document.getElementById('inactive-body');
    if (!inactive.length) {
      tbody.innerHTML = '<tr><td colspan="4" style="text-align:center;color:#22c55e;padding:16px;">✅ 所有用户7天内均有活动</td></tr>';
      return;
    }
    tbody.innerHTML = inactive.map(([uid, info]) => {
      const daysAgo = Math.floor((Date.now() - new Date(info.ts)) / 86400000);
      const cls = daysAgo >= 30 ? 'badge-danger' : daysAgo >= 14 ? 'badge-warn' : 'badge-ok';
      return `<tr>
        <td><a class="user-link" onclick="showUserDetail('${uid}')">${info.alias}</a></td>
        <td class="metric-value">${daysAgo}天</td>
        <td style="color:#475569;font-size:0.75rem;">${new Date(info.ts).toLocaleDateString('zh-CN')}</td>
        <td><span class="badge ${cls}">${daysAgo >= 30 ? '紧急' : daysAgo >= 14 ? '注意' : '监控中'}</span></td>
      </tr>`;
    }).join('');
  }

  function renderInsights(successRate, p95Latency, totalCost) {
    const insights = [];
    const sevenDaysAgo = new Date(Date.now() - 7 * 86400000);
    const activeUsers = new Set(allData.filter(r => new Date(r.timestamp) > sevenDaysAgo).map(r => r.user_id));
    const totalUsers = new Set(allData.map(r => r.user_id)).size;
    const adoptionRate = totalUsers ? Math.round(activeUsers.size / totalUsers * 100) : 0;
    const inactiveCount = totalUsers - activeUsers.size;

    if (inactiveCount > 0) insights.push(`■ 非活跃用户 <strong>${inactiveCount}名</strong> — 建议考虑LLM使用引导与支持`);
    if (successRate < 95) insights.push(`■ 成功率 ${successRate}% → 未达SLA 95%要求 ⚠️ 需分析错误原因`);
    if (p95Latency > 2000) insights.push(`■ p95 延迟 ${p95Latency}ms → 超出SLA要求 — 可考虑轻量化模型`);
    if (adoptionRate < 80) insights.push(`▲ 团队普及率 ${adoptionRate}% → 未达80%目标 (${activeUsers.size}/${totalUsers}名活跃)`);
    if (totalCost > 50) insights.push(`▲ 总成本 $${totalCost} — 建议高成本用户考虑模型优化`);

    const categories = {};
    allData.forEach(r => { categories[r.prompt_category || '其他'] = (categories[r.prompt_category || '其他'] || 0) + 1; });
    const topCat = Object.entries(categories).sort((a,b) => b[1]-a[1])[0];
    if (topCat) insights.push(`● 主要使用场景: <strong>${topCat[0]}</strong> (${topCat[1]}次) — 引入专用模型效果更佳`);

    const insightDiv = document.getElementById('pm-insights');
    insightDiv.innerHTML = `<div class="insight-box">
      <h4>💡 PM自动洞察 — 截至${new Date().toLocaleDateString('zh-CN')}</h4>
      <ul>${insights.map(i => `<li>${i}</li>`).join('')}</ul>
    </div>`;
  }

  function showUserDetail(userId) {
    const u = allUsers[userId];
    if (!u) return;
    const userRuns = allData.filter(r => r.user_id === userId);
    const categories = {};
    userRuns.forEach(r => { categories[r.prompt_category || '其他'] = (categories[r.prompt_category || '其他'] || 0) + 1; });
    const totalCost = userRuns.reduce((s, r) => s + (r.cost_usd || 0), 0).toFixed(4);
    const topModel = Object.entries(
      userRuns.reduce((m, r) => { m[r.model] = (m[r.model] || 0)+1; return m; }, {})
    ).sort((a,b) => b[1]-a[1])[0]?.[0] || '-';

    document.getElementById('user-detail-content').innerHTML = `
      <div style="background:var(--bg-elevated);border-radius:8px;padding:16px;margin-bottom:20px;">
        <h2 style="font-size:1.25rem;margin-bottom:8px;">👤 ${u.alias}</h2>
        <div style="display:grid;grid-template-columns:repeat(4,1fr);gap:12px;margin-top:12px;">
          <div><div style="font-size:0.625rem;color:#475569;text-transform:uppercase;margin-bottom:4px;">总成本</div><div class="metric-value" style="font-size:1.5rem;">$${totalCost}</div></div>
          <div><div style="font-size:0.625rem;color:#475569;text-transform:uppercase;margin-bottom:4px;">总请求</div><div class="metric-value" style="font-size:1.5rem;">${u.runs.toLocaleString()}</div></div>
          <div><div style="font-size:0.625rem;color:#475569;text-transform:uppercase;margin-bottom:4px;">偏好模型</div><div style="font-size:1rem;margin-top:4px;">${topModel}</div></div>
          <div><div style="font-size:0.625rem;color:#475569;text-transform:uppercase;margin-bottom:4px;">分类分布</div><div style="font-size:0.8rem;color:#94a3b8;">${Object.entries(categories).map(([k,v]) => k+' '+v+'次').join(', ')}</div></div>
        </div>
      </div>
      <h3 style="font-size:0.875rem;color:#94a3b8;margin-bottom:12px;">最近运行日志</h3>
      <table class="ranking-table">
        <thead><tr><th>时间</th><th>模型</th><th>分类</th><th>成本</th><th>延迟</th><th>状态</th></tr></thead>
        <tbody>
          ${userRuns.slice(-10).reverse().map(r => {
            const sc = r.status_code === 200 ? 'badge-ok' : 'badge-danger';
            return `<tr>
              <td style="color:#475569;font-size:0.75rem;">${new Date(r.timestamp).toLocaleString('zh-CN')}</td>
              <td style="font-size:0.8rem;">${r.model}</td>
              <td><span class="badge badge-ok" style="font-size:0.65rem;">${r.prompt_category||'其他'}</span></td>
              <td class="metric-value">$${(r.cost_usd||0).toFixed(4)}</td>
              <td class="metric-value">${(r.latency_avg_ms||0).toFixed(0)}ms</td>
              <td><span class="badge ${sc}">${r.status_code||200}</span></td>
            </tr>`;
          }).join('')}
        </tbody>
      </table>
      <div class="insight-box" style="margin-top:16px;">
        <h4>💡 个人洞察</h4>
        <ul>
          <li>偏好模型: <strong>${topModel}</strong> — 切换为同性能轻量化模型可降低成本</li>
          <li>主要使用场景: <strong>${Object.entries(categories).sort((a,b)=>b[1]-a[1])[0]?.[0]||'无'}</strong></li>
          <li>共运行${u.runs}次 — 需对比团队平均水平分析活跃度</li>
        </ul>
      </div>
    `;
    document.getElementById('main-dashboard').style.display = 'none';
    document.getElementById('user-detail').style.display = 'block';
    window.scrollTo(0, 0);
  }

  function showMain() {
    document.getElementById('user-detail').style.display = 'none';
    document.getElementById('main-dashboard').style.display = 'block';
  }

  // 键盘快捷键
  document.addEventListener('keydown', e => {
    if (e.key === 'r' || e.key === 'R') loadData();
    if (e.key === 'Escape') showMain();
  });

  // 初始加载
  loadData();
  // 每5分钟自动刷新
  setInterval(loadData, 5 * 60 * 1000);
  </script>
</body>
</html>
HTML_EOF

echo "✅ 轻量HTML仪表盘生成完成: llm-monitoring/index.html"
# 启动本地服务器
cd llm-monitoring && python3 -m http.server "${DASHBOARD_PORT:-3000}" &
echo "✅ 仪表盘运行中: http://localhost:${DASHBOARD_PORT:-3000}"

Step 4: PM洞察Tab和排名系统

（Option A / Next.js 适用）

bash

# 创建PM仪表盘API路由
cat > app/api/ranking/route.ts << 'TS_EOF'
import { NextRequest, NextResponse } from 'next/server'
import db from '@/lib/llm-monitoring/db'

export async function GET(req: NextRequest) {
  const period = req.nextUrl.searchParams.get('period') || '30d'
  const days   = period === '7d' ? 7 : period === '90d' ? 90 : 30

  // 按成本排名
  const costRanking = db.prepare(`
    SELECT
      user_id, user_alias,
      ROUND(SUM(cost_usd), 4)           AS total_cost,
      COUNT(*)                           AS total_runs,
      GROUP_CONCAT(DISTINCT model)       AS models_used,
      ROUND(AVG(latency_avg_ms), 0)      AS avg_latency,
      ROUND(
        AVG(CAST(successful AS REAL) / NULLIF(total_requests, 0)) * 100, 1
      )                                  AS success_rate,
      MAX(timestamp)                     AS last_seen
    FROM runs
    WHERE timestamp >= datetime('now', '-' || ? || ' days')
    GROUP BY user_id
    ORDER BY total_cost DESC
    LIMIT 20
  `).all(days)

  // 非活跃用户追踪（所选时间段内无活动的注册用户）
  const inactiveUsers = db.prepare(`
    SELECT
      p.user_id, p.user_alias, p.team,
      MAX(r.timestamp)  AS last_seen,
      CAST((julianday('now') - julianday(MAX(r.timestamp))) AS INTEGER) AS days_inactive
    FROM user_profiles p
    LEFT JOIN runs r ON p.user_id = r.user_id
    GROUP BY p.user_id
    HAVING last_seen IS NULL
       OR days_inactive >= 7
    ORDER BY days_inactive DESC
  `).all()

  // PM摘要
  const summary = db.prepare(`
    SELECT
      COUNT(DISTINCT user_id)    AS total_users,
      COUNT(DISTINCT CASE WHEN timestamp >= datetime('now', '-7 days') THEN user_id END) AS active_7d,
      ROUND(SUM(cost_usd), 2)    AS total_cost,
      COUNT(*)                   AS total_runs
    FROM runs
    WHERE timestamp >= datetime('now', '-' || ? || ' days')
  `).get(days) as Record<string, number>

  return NextResponse.json({ costRanking, inactiveUsers, summary })
}
TS_EOF

Step 5: 自动生成PM周报

bash

cat > generate-pm-report.sh << 'REPORT_EOF'
#!/usr/bin/env bash
# generate-pm-report.sh — 自动生成PM周报（Markdown格式）
set -euo pipefail

REPORT_DATE=$(date +"%Y-%m-%d")
REPORT_WEEK=$(date +"%Y-W%V")
OUTPUT_DIR="./reports"
OUTPUT="${OUTPUT_DIR}/pm-weekly-${REPORT_DATE}.md"
mkdir -p "$OUTPUT_DIR"

python3 << PYEOF > "$OUTPUT"
import json, sys
from datetime import datetime, timedelta
from collections import defaultdict

# 加载最近7天数据
try:
    records = [json.loads(l) for l in open('./data/metrics.jsonl') if l.strip()]
except FileNotFoundError:
    records = []

week_ago = (datetime.now() - timedelta(days=7)).isoformat()
week_data = [r for r in records if r.get('timestamp', '') >= week_ago]

# 统计
total_cost    = sum(r.get('cost_usd', 0) for r in week_data)
total_runs    = len(week_data)
active_users  = set(r['user_id'] for r in week_data)
all_users     = set(r['user_id'] for r in records)
inactive_users = all_users - active_users

# 按用户成本排名
user_costs = defaultdict(lambda: {'cost': 0, 'runs': 0, 'alias': '', 'categories': defaultdict(int)})
for r in week_data:
    uid = r.get('user_id', 'unknown')
    user_costs[uid]['cost']  += r.get('cost_usd', 0)
    user_costs[uid]['runs']  += 1
    user_costs[uid]['alias']  = r.get('user_alias', uid)
    user_costs[uid]['categories'][r.get('prompt_category', '其他')] += 1

top_users = sorted(user_costs.items(), key=lambda x: x[1]['cost'], reverse=True)[:5]

# 按模型使用量
model_usage = defaultdict(int)
for r in week_data:
    model_usage[r.get('model', 'unknown')] += 1
top_model = max(model_usage, key=model_usage.get) if model_usage else '-'

# 成功率
success_count = sum(1 for r in week_data if r.get('status_code', 200) == 200)
success_rate  = (success_count / total_runs * 100) if total_runs else 0

print(f"""# 📊 LLM使用周报 — {REPORT_DATE} ({REPORT_WEEK})

## 执行摘要

| 指标 | 值 |
|------|-----|
| 总成本 | \${total_cost:.2f} |
| 总运行次数 | {total_runs:,}次 |
| 活跃用户 | {len(active_users)}名 |
| 普及率 | {len(active_users)}/{len(all_users)}名 ({len(active_users)/len(all_users)*100:.0f}% if all_users else 'N/A') |
| 成功率 | {success_rate:.1f}% |
| 使用最多的模型 | {top_model} |

## 🏆 TOP 5用户（按成本）

| 排名 | 用户 | 成本 | 运行次数 | 主要分类 |
|------|--------|------|---------|--------------|
{"".join(f"| {'🥇🥈🥉'[i] if i < 3 else i+1} | {u['alias']} | \${u['cost']:.4f} | {u['runs']} | {max(u['categories'], key=u['categories'].get) if u['categories'] else '-'} |" + chr(10) for i, (uid, u) in enumerate(top_users))}

## 💤 非活跃用户 ({len(inactive_users)}名)

{"无 — 所有用户7天内均有活动" if not inactive_users else chr(10).join(f"- {uid}" for uid in inactive_users)}

## 💡 PM建议措施

{"- 针对" + str(len(inactive_users)) + "名非活跃用户开展引导与支持" if inactive_users else ""}
{"- 成功率 " + f"{success_rate:.1f}%" + " — SLA 95% " + ("达标 ✅" if success_rate >= 95 else "未达标 ⚠️ 需分析错误原因") }
{"- 总成本 \$" + f"{total_cost:.2f}" + " — 可对比上周数据寻找模型优化机会"}

---
*自动生成: generate-pm-report.sh | 基于Tokuin CLI*
""")
PYEOF

echo "✅ PM报告已生成: $OUTPUT"
cat "$OUTPUT"

# Slack通知（已配置的情况下）
if [ -n "${SLACK_WEBHOOK_URL:-}" ]; then
  SUMMARY=$(grep -A5 "## 执行摘要" "$OUTPUT" | tail -5)
  curl -s -X POST "$SLACK_WEBHOOK_URL" \
    -H 'Content-type: application/json' \
    -d "{\"text\":\"📊 LLM周度报告 ($REPORT_DATE)\n$SUMMARY\"}" > /dev/null
  echo "✅ Slack通知发送完成"
fi
REPORT_EOF
chmod +x generate-pm-report.sh

# 每周一上午9点自动运行
(crontab -l 2>/dev/null; echo "0 9 * * 1 cd $(pwd) && bash generate-pm-report.sh >> ./data/report.log 2>&1") | crontab -
echo "✅ 周报cron定时任务注册完成（每周一 09:00）"

# 立即测试运行
bash generate-pm-report.sh

Step 6: 配置成本告警

bash

cat > check-alerts.sh << 'ALERT_EOF'
#!/usr/bin/env bash
# check-alerts.sh — 检测成本阈值超出发送Slack通知
set -euo pipefail

THRESHOLD="${COST_THRESHOLD_USD:-10.00}"

CURRENT_COST=$(python3 << PYEOF
import json
from datetime import datetime, timedelta

today = datetime.now().date().isoformat()
try:
    records = [json.loads(l) for l in open('./data/metrics.jsonl') if l.strip()]
    today_cost = sum(r.get('cost_usd', 0) for r in records if r.get('timestamp', '')[:10] == today)
    print(f"{today_cost:.4f}")
except:
    print("0.0000")
PYEOF
)

python3 - << PYEOF
import sys
cost, threshold = float('$CURRENT_COST'), float('$THRESHOLD')
if cost > threshold:
    print(f"ALERT: 今日成本 \${cost:.4f} 超过阈值 \${threshold:.2f}!")
    sys.exit(1)
else:
    print(f"正常: 今日成本 \${cost:.4f} / 阈值 \${threshold:.2f}")
    sys.exit(0)
PYEOF

# exit 1时发送Slack通知
if [ $? -ne 0 ] && [ -n "${SLACK_WEBHOOK_URL:-}" ]; then
  curl -s -X POST "$SLACK_WEBHOOK_URL" \
    -H 'Content-type: application/json' \
    -d "{\"text\":\"⚠️ LLM成本超出阈值!\n今日成本: \$$CURRENT_COST / 阈值: \$$THRESHOLD\"}" > /dev/null
fi
ALERT_EOF
chmod +x check-alerts.sh

# 每小时检查一次成本
(crontab -l 2>/dev/null; echo "0 * * * * cd $(pwd) && bash check-alerts.sh >> ./data/alerts.log 2>&1") | crontab -
echo "✅ 成本告警cron定时任务注册完成（每小时运行一次）"

隐私政策

yaml

# 隐私保护政策（必须遵守）

prompt_storage:
  store_full_prompt: false      # 默认值: 不存储提示词原文
  store_preview:     false      # 默认也不存储前100字符预览（需管理员明确配置开启）
  store_hash:        true       # 仅存储SHA-256哈希（用于模式分析）

user_data:
  anonymize_by_default: true   # user_id默认可用哈希存储（可通过LLM_USER_ID环境变量控制）
  retention_days: 90            # 建议90天后清理旧数据
  
compliance:
  # 绝对禁止将API密钥写入代码/HTML/日志文件
  # .env文件必须加入.gitignore
  # 限制非管理员访问提示词预览功能

⚠️ 开启
store_preview: true
的必备流程
存储提示词预览仅可在管理员明确完成以下流程后开启:
在
.env
文件中设置
STORE_PREVIEW=true
（禁止直接修改代码）
获得团队成员的个人信息处理同意（告知用户将存储提示词预览）

访问权限仅开放给管理员角色（普通用户不可查看）
明确设置
retention_days
指定存储期限
未完成以上流程开启
store_preview: true
属于严格禁止的违规行为。

输出格式

技能执行完成后生成的文件:

./
├── safety-guard.sh          # 安全检查脚本（Step 0）
├── categorize_prompt.py     # 提示词自动分类
├── collect-metrics.sh       # 指标采集（Step 2）
├── generate-pm-report.sh    # PM周报生成（Step 5）
├── check-alerts.sh          # 成本告警（Step 6）
│
├── data/
│   ├── metrics.jsonl        # 时序指标（JSONL格式）
│   ├── collect.log          # 采集日志
│   ├── alerts.log           # 告警日志
│   └── reports/
│       └── pm-weekly-YYYY-MM-DD.md  # 自动生成的PM报告
│
├── [选择Next.js时生成]
│   ├── app/admin/llm-monitoring/page.tsx
│   ├── app/admin/llm-monitoring/users/[userId]/page.tsx
│   ├── app/api/runs/route.ts
│   ├── app/api/ranking/route.ts
│   ├── app/api/metrics/route.ts        # Prometheus端点
│   ├── components/llm-monitoring/
│   │   ├── KPICard.tsx
│   │   ├── TrendChart.tsx
│   │   ├── ModelCostBar.tsx
│   │   ├── LatencyGauge.tsx
│   │   ├── TokenDonut.tsx
│   │   ├── RankingTable.tsx
│   │   ├── InactiveUsers.tsx
│   │   ├── PMInsights.tsx
│   │   └── UserDetailPage.tsx
│   └── lib/llm-monitoring/db.ts
│
└── [选择轻量HTML时生成]
    └── llm-monitoring/
        ├── index.html       # 单文件仪表盘（含图表+排名+用户详情）
        └── data/
            └── metrics.jsonl

约束规则

MUST（必须遵守）

必须最先执行Step 0(
safety-guard.sh
)
默认使用
```
--dry-run
```
模式，真实API调用必须明确指定
```
--allow-live
```
参数
API密钥必须通过环境变量或
```
.env
```
文件管理
必须将
```
.env
```
加入.gitignore:
```
echo '.env' >> .gitignore
```
状态显示必须统一使用三级颜色系统(
```
--color-ok
```
,
```
--color-warn
```
,
```
--color-danger
```
)
必须实现用户链接点击跳转到个人详情页的下钻导航功能
PM洞察必须基于数据自动生成（禁止硬编码）

MUST NOT（严格禁止）

禁止将API密钥直接写入代码、HTML、脚本、日志文件
禁止将真实API调用(
```
--allow-live
```
)设为自动化脚本的默认模式
禁止使用任意自定义颜色 — 必须仅使用设计token CSS变量
禁止仅用文本显示状态（必须同时使用颜色+徽章）
禁止在数据库中存储用户提示词原文（仅允许存储哈希）

使用示例

示例1: 快速开始（dry-run，无需API密钥）

bash

# 1. 安全检查
bash safety-guard.sh

# 2. 安装Tokuin
curl -fsSL https://raw.githubusercontent.com/nooscraft/tokuin/main/install.sh | bash

# 3. 采集样例数据（dry-run）
export LLM_USER_ID="dev-alice"
export LLM_USER_ALIAS="Alice"
bash collect-metrics.sh "Analyze user behavior patterns"
bash collect-metrics.sh "Write a Python function to parse JSON"
bash collect-metrics.sh "Translate this document to English"

# 4. 启动轻量仪表盘
cd llm-monitoring && python3 -m http.server 3000
open http://localhost:3000

示例2: 多用户模拟（团队测试）

bash

# 模拟多个用户的dry-run
for user in "alice" "backend" "analyst" "pm-charlie"; do
  export LLM_USER_ID="$user"
  export LLM_USER_ALIAS="$user"
  for category in "编码" "分析" "翻译"; do
    bash collect-metrics.sh "${category}相关提示词示例"
  done
done

# 查看结果
wc -l data/metrics.jsonl

示例3: 立即生成PM周报

bash

bash generate-pm-report.sh
cat reports/pm-weekly-$(date +%Y-%m-%d).md

示例4: 测试成本告警

bash

export COST_THRESHOLD_USD=0.01   # 用低阈值测试
bash check-alerts.sh
# 预期输出: 超过阈值输出ALERT，否则输出"正常"

参考资料

Tokuin GitHub: https://github.com/nooscraft/tokuin
Tokuin安装脚本: https://raw.githubusercontent.com/nooscraft/tokuin/main/install.sh
添加模型指南: https://github.com/nooscraft/tokuin/blob/main/ADDING_MODELS_GUIDE.md
提供商路线图: https://github.com/nooscraft/tokuin/blob/main/PROVIDERS_PLAN.md
贡献指南: https://github.com/nooscraft/tokuin/blob/main/CONTRIBUTING.md
OpenRouter模型目录: https://openrouter.ai/models
韩语博客指南: https://digitalbourgeois.tistory.com/m/2658

llm-monitoring-dashboard

NPX安装

标签

SKILL.md 内容（中文）

LLM使用监控仪表盘

本技能适用场景

前置条件

1. Tokuin CLI 安装确认

2. 环境变量（仅实际调用API时需要）

3. 项目技术栈要求

使用指南

Step 0: 安全检查（必须最先执行）

Step 1: 安装Tokuin CLI并验证dry-run

Step 2: 包含用户上下文的数据采集管道

Step 3: 创建路由结构和仪表盘框架

Step 4: PM洞察Tab和排名系统

Step 5: 自动生成PM周报

Step 6: 配置成本告警

隐私政策

输出格式

约束规则

MUST（必须遵守）

MUST NOT（严格禁止）

使用示例

示例1: 快速开始（dry-run，无需API密钥）

示例2: 多用户模拟（团队测试）

示例3: 立即生成PM周报

示例4: 测试成本告警

参考资料