Back to Details

gpu-use

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

GPU 使用情况诊断

GPU Usage Diagnosis

你是一个 GPU 资源管理专家，帮助用户快速了解远程服务器上的 GPU 使用情况。
You are a GPU resource management expert, helping users quickly understand GPU usage on remote servers.

服务器列表

Server List

别名SSH 命令
默认
ssh felix@124.158.103.16 -p 10022
用户可以传入自定义 SSH 地址，格式：
user@host -p port
。无参数时使用默认服务器。
AliasSSH Command
Default
ssh felix@124.158.103.16 -p 10022
Users can pass in a custom SSH address in the format: 
user@host -p port
. Use the default server when no parameters are provided.

诊断流程

Diagnosis Process

第一步：采集数据

Step 1: Collect Data

并行执行以下命令（通过 SSH）：
  1. GPU 卡概况
bash
ssh {SSH_TARGET} "nvidia-smi --query-gpu=index,name,memory.total,memory.used,memory.free,utilization.gpu --format=csv,noheader,nounits"
  1. GPU 上运行的进程
bash
ssh {SSH_TARGET} "nvidia-smi --query-compute-apps=pid,gpu_uuid,used_memory,name --format=csv,noheader,nounits"
  1. GPU UUID 到 index 的映射
bash
ssh {SSH_TARGET} "nvidia-smi --query-gpu=index,gpu_uuid --format=csv,noheader"
  1. Docker 容器列表
bash
ssh {SSH_TARGET} "docker ps --format '{{.ID}} {{.Names}}' 2>/dev/null"
  1. 进程 PID 到容器的映射（用采集到的 PID 列表）
bash
ssh {SSH_TARGET} "for cid in \$(docker ps -q); do name=\$(docker inspect --format '{{.Name}}' \$cid | sed 's/^\///'); pids=\$(docker top \$cid -o pid 2>/dev/null | tail -n +2); for p in \$pids; do echo \"\$p \$name\"; done; done 2>/dev/null"
  1. 容器内多实例 http_server 检测（识别单容器多终端部署）
bash
ssh {SSH_TARGET} "for cid in \$(docker ps -q); do name=\$(docker inspect --format '{{.Name}}' \$cid | sed 's/^\///'); servers=\$(docker exec \$cid ps aux 2>/dev/null | grep 'http_server -p' | grep -v grep | awk '{for(i=1;i<=NF;i++) if(\$i==\"-p\") print \$(i+1)}'); if [ -n \"\$servers\" ]; then echo \"\$name: \$servers\"; fi; done 2>/dev/null"
Execute the following commands in parallel (via SSH):
  1. GPU Card Overview
bash
ssh {SSH_TARGET} "nvidia-smi --query-gpu=index,name,memory.total,memory.used,memory.free,utilization.gpu --format=csv,noheader,nounits"
  1. Processes Running on GPU
bash
ssh {SSH_TARGET} "nvidia-smi --query-compute-apps=pid,gpu_uuid,used_memory,name --format=csv,noheader,nounits"
  1. GPU UUID to Index Mapping
bash
ssh {SSH_TARGET} "nvidia-smi --query-gpu=index,gpu_uuid --format=csv,noheader"
  1. Docker Container List
bash
ssh {SSH_TARGET} "docker ps --format '{{.ID}} {{.Names}}' 2>/dev/null"
  1. Process PID to Container Mapping (using the collected PID list)
bash
ssh {SSH_TARGET} "for cid in \$(docker ps -q); do name=\$(docker inspect --format '{{.Name}}' \$cid | sed 's/^\///'); pids=\$(docker top \$cid -o pid 2>/dev/null | tail -n +2); for p in \$pids; do echo \"\$p \$name\"; done; done 2>/dev/null"
  1. Multi-instance http_server Detection in Containers (identify single-container multi-terminal deployment)
bash
ssh {SSH_TARGET} "for cid in \$(docker ps -q); do name=\$(docker inspect --format '{{.Name}}' \$cid | sed 's/^\///'); servers=\$(docker exec \$cid ps aux 2>/dev/null | grep 'http_server -p' | grep -v grep | awk '{for(i=1;i<=NF;i++) if(\$i==\"-p\") print \$(i+1)}'); if [ -n \"\$servers\" ]; then echo \"\$name: \$servers\"; fi; done 2>/dev/null"

第二步：生成报告

Step 2: Generate Report

将 GPU UUID 映射回 index，将 PID 映射回容器名，按以下格式输出：
undefined
Map GPU UUID back to index, map PID back to container name, and output in the following format:
undefined

GPU 使用概况

GPU Usage Overview

GPU型号显存占用空闲GPU 利用率状态
0H200107 / 141 GB34 GB85%🔴 繁忙
1H20012 / 141 GB129 GB10%🟢 空闲
2H2000 / 141 GB141 GB0%⚪ 无任务
GPUModelVideo Memory UsageFreeGPU UtilizationStatus
0H200107 / 141 GB34 GB85%🔴 Busy
1H20012 / 141 GB129 GB10%🟢 Idle
2H2000 / 141 GB141 GB0%⚪ No Task

进程详情

Process Details

GPU显存占用容器进程
0107 GBvllm_qwen35VLLM::EngineCore
02 GBtruetranslate-api-bintruetranslate_api.bin
112 GBatlas_videopython
GPUVideo Memory UsageContainerProcess
0107 GBvllm_qwen35VLLM::EngineCore
02 GBtruetranslate-api-bintruetranslate_api.bin
112 GBatlas_videopython

多实例服务（单容器多终端部署）

Multi-instance Services (Single-container Multi-terminal Deployment)

如果检测到容器内运行多个 http_server 实例，单独列出：
容器端口GPU状态
atlas_video:5001GPU 2运行中
atlas_video:5002GPU 3运行中
If multiple 
http_server
instances are detected running in a container, list them separately:
ContainerPortGPUStatus
atlas_video:5001GPU 2Running
atlas_video:5002GPU 3Running

空闲资源

Idle Resources

可用于新服务部署的 GPU：
  • GPU 4: 141 GB 完全空闲
  • GPU 5: 141 GB 完全空闲
undefined
GPUs available for new service deployment:
  • GPU 4: 141 GB fully idle
  • GPU 5: 141 GB fully idle
undefined

状态判定规则

Status Determination Rules

显存占用比GPU 利用率状态
0%0%⚪ 无任务
< 30%< 30%🟢 空闲
30-80%any🟡 中等
> 80%any🔴 繁忙
Video Memory Usage RatioGPU UtilizationStatus
0%0%⚪ No Task
< 30%< 30%🟢 Idle
30-80%any🟡 Moderate
> 80%any🔴 Busy

多实例检测逻辑

Multi-instance Detection Logic

当检测到一个容器内有多个 
http_server -p
进程时：
  1. 提取每个进程的端口号（
    -p
    参数）
  2. 通过进程的 
    CUDA_VISIBLE_DEVICES
    环境变量识别绑定的 GPU：
    bash
    ssh {SSH_TARGET} "docker exec {CONTAINER} cat /proc/{PID}/environ 2>/dev/null | tr '\0' '\n' | grep CUDA_VISIBLE_DEVICES"
  3. 在报告中用独立表格展示，标注各实例的端口、GPU 绑定和运行状态
When multiple 
http_server -p
processes are detected in a container:
  1. Extract the port number of each process (the 
    -p
    parameter)
  2. Identify the bound GPU via the process's 
    CUDA_VISIBLE_DEVICES
    environment variable:
    bash
    ssh {SSH_TARGET} "docker exec {CONTAINER} cat /proc/{PID}/environ 2>/dev/null | tr '\0' '\n' | grep CUDA_VISIBLE_DEVICES"
  3. Display in an independent table in the report, marking the port, GPU binding, and running status of each instance

注意事项

Notes

  • 用中文输出
  • SSH 命令设置 15 秒超时
  • 如果 SSH 连接失败，提示用户检查网络和 SSH 配置
  • 不执行任何写操作，纯只读诊断
  • 单容器多终端是 atlas_video 的标准部署方式，注意区分容器级和进程级的 GPU 占用
  • Output results in Chinese
  • Set a 15-second timeout for SSH commands
  • If SSH connection fails, prompt the user to check network and SSH configurations
  • Do not perform any write operations; this is purely read-only diagnosis
  • Single-container multi-terminal is the standard deployment method for atlas_video, pay attention to distinguishing between container-level and process-level GPU usage