huawei-cloud-msmodelslim-model-adapt

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Huawei Cloud msModelSlim Model Adapter

华为云msModelSlim模型适配器

Overview

概述

This skill guides how to create basic adapters for new models to run W8A8/W4A16 quantization workflows in msModelSlim.

Architecture: Model Analysis -> Adapter Creation -> Registration -> Verification (4 Steps)

Related Skills:

```
huawei-cloud-msmodelslim-model-analysis
```
- Model structure analysis before adapter implementation
```
huawei-cloud-ascend-profiler-db-explorer
```
- Optional: Performance analysis after deployment

本技能指导如何为新模型创建基础适配器，以在msModelSlim中运行W8A8/W4A16量化流程。

架构：模型分析 -> 适配器创建 -> 注册 -> 验证（四步）

相关技能：

```
huawei-cloud-msmodelslim-model-analysis
```
- 适配器实现前的模型结构分析

huawei-cloud-ascend-profiler-db-explorer

- 可选：部署后的性能分析

Scope

适用范围

Supported:

Decoder-only LLM
Understanding VLM (text/LLM backbone only)

Not supported:

Multimodal generation (Stable Diffusion/Flux/Wan)
Encoder-only models
Non-Transformers architectures

支持的模型类型：

仅解码器LLM
理解型VLM（仅文本/LLM骨干网络）

不支持的模型类型：

多模态生成模型（Stable Diffusion/Flux/Wan）
仅编码器模型
非Transformers架构模型

Architecture

架构

text

┌─────────────────────────────────────────────────────────────┐
│              msModelSlim Model Adapter Skill                │
├─────────────────────────────────────────────────────────────┤
│  ┌──────────────────┐    ┌──────────────────────────────┐  │
│  │  Model Analysis │───▶│    Adapter Creation          │  │
│  │  - config.json  │    │    - LLM Adapter Template     │  │
│  │  - modeling_*.py│    │    - VLM Adapter Template     │  │
│  └──────────────────┘    │    - Required Interfaces     │  │
│                          └──────────────────────────────┘  │
│                                    │                       │
│                                    ▼                       │
│                          ┌──────────────────┐             │
│                          │  Registration    │             │
│                          │  & Installation  │             │
│                          └──────────────────┘             │
│                                    │                       │
│                                    ▼                       │
│  ┌──────────────────────────────────────────────────────┐ │
│  │                   Verification (4 Steps)              │ │
│  │  1. Generate Test Model → 2. Full Fallback Quant     │ │
│  │  3. Weight Verification → 4. Quant Description     │ │
│  └──────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────┘

text

┌─────────────────────────────────────────────────────────────┐
│              msModelSlim Model Adapter Skill                │
├─────────────────────────────────────────────────────────────┤
│  ┌──────────────────┐    ┌──────────────────────────────┐  │
│  │  Model Analysis │───▶│    Adapter Creation          │  │
│  │  - config.json  │    │    - LLM Adapter Template     │  │
│  │  - modeling_*.py│    │    - VLM Adapter Template     │  │
│  └──────────────────┘    │    - Required Interfaces     │  │
│                          └──────────────────────────────┘  │
│                                    │                       │
│                                    ▼                       │
│                          ┌──────────────────┐             │
│                          │  Registration    │             │
│                          │  & Installation  │             │
│                          └──────────────────┘             │
│                                    │                       │
│                                    ▼                       │
│  ┌──────────────────────────────────────────────────────┐ │
│  │                   Verification (4 Steps)              │ │
│  │  1. Generate Test Model → 2. Full Fallback Quant     │ │
│  │  3. Weight Verification → 4. Quant Description     │ │
│  └──────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────┘

Architecture Components

架构组件

This skill involves the following cloud services and components:

msModelSlim: Huawei Cloud's model quantization framework for efficient model compression
Transformers Library: Hugging Face Transformers for model loading and processing
ModelScope: Model download and management platform
Ascend NPU: Target hardware for quantized model deployment

本技能涉及以下云服务和组件：

msModelSlim：华为云的模型量化框架，用于高效模型压缩
Transformers Library：Hugging Face Transformers，用于模型加载和处理
ModelScope：模型下载与管理平台
Ascend NPU：量化模型部署的目标硬件

Use Cases

使用场景

Typical Problem Scenarios:

Need to deploy LLM models with reduced memory footprint on Ascend NPU
Want to optimize inference speed without significant accuracy loss
Migrating models that don't have built-in msModelSlim support
Need W8A8/W4A16 quantization for decoder-only LLM or VLM text backbones

Typical User Phrases:

"How to quantize my custom LLM model for Ascend?"
"Create msModelSlim adapter for Qwen model"
"Implement W4A16 quantization workflow"
"Adapt my VLM text backbone for quantization"
"How to add quantization support for new models?"

典型问题场景：

需要在Ascend NPU上部署内存占用更低的LLM模型
希望在不显著损失精度的前提下优化推理速度
迁移不具备内置msModelSlim支持的模型
需要为仅解码器LLM或VLM文本骨干网络实现W8A8/W4A16量化

典型用户提问：

"如何为Ascend量化我的自定义LLM模型？"
"为Qwen模型创建msModelSlim适配器"
"实现W4A16量化流程"
"适配我的VLM文本骨干网络以支持量化"
"如何为新模型添加量化支持？"

Core Workflow

核心流程

1. Preparation

1. 准备工作

Download Model: Recommended to use

modelscope download

for non-weight files.

Example:

modelscope download --model <org>/<model> --local_dir ./models/<name> --exclude '*.safetensors'

Analyze Model: Read
```
config.json
```
and
```
modeling_*.py
```
to confirm structure and implementation.
- See: Model Analysis Guide

下载模型：推荐使用

modelscope download

下载非权重文件。

示例：

modelscope download --model <org>/<model> --local_dir ./models/<name> --exclude '*.safetensors'

分析模型：读取
```
config.json
```
和
```
modeling_*.py
```
以确认模型结构和实现方式。
- 参考：模型分析指南

2. Create Adapter

2. 创建适配器

Use Templates:

LLM:
```
assets/model_adapter_template.py
```
VLM:
```
assets/vlm_model_adapter_template.py
```

Implement Interfaces: Implement

handle_dataset

init_model

generate_model_visit

generate_model_forward

enable_kv_cache

Key Principles:
- ```
visit
```
  and
```
forward
```
  must be strictly consistent.
- MoE models recommended to unpack to pure linear layers.
- See: Implementation Guide

使用模板：

LLM：
```
assets/model_adapter_template.py
```
VLM：
```
assets/vlm_model_adapter_template.py
```

实现接口：实现

handle_dataset

init_model

generate_model_visit

generate_model_forward

enable_kv_cache

接口。

核心原则：
- ```
visit
```
  和
```
forward
```
  必须严格保持一致。
- 建议将MoE模型拆分为纯线性层。
- 参考：实现指南

3. Registration & Installation

3. 注册与安装

Register model and entry in
```
config/config.ini
```
, then execute
```
bash install.sh
```
.
See: Registration Guide

在
```
config/config.ini
```
中注册模型和入口，然后执行
```
bash install.sh
```
。
参考：注册指南

4. Verify Adapter (Required)

4. 验证适配器（必填）

Must execute four-step verification: Generate test model -> Full fallback quantization -> Verify full fallback model matches float weights exactly and can load/save completely -> Verify actual quantization workflow works (including description file rule validation).
See: Verification Guide

必须执行四步验证：生成测试模型 -> 全回退量化 -> 验证全回退模型与浮点权重完全匹配且可完整加载/保存 -> 验证实际量化流程正常运行（包括描述文件规则验证）。
参考：验证指南

Common Scripts

常用脚本

Scripts located in

scripts/

directory:

```
scripts/step1_generate_test_model.py
```
```
scripts/step2_run_quantization.py
```
```
scripts/step3_verify_weights.py
```

scripts/step4_verify_quant_description.py

脚本位于

scripts/

目录下：

```
scripts/step1_generate_test_model.py
```
```
scripts/step2_run_quantization.py
```
```
scripts/step3_verify_weights.py
```

scripts/step4_verify_quant_description.py

Prerequisites

前置条件

System Requirements

系统要求

Python 3.8+
transformers >= 4.40.0
msmodelslim >= 1.0.0

Python 3.8+
transformers >= 4.40.0
msmodelslim >= 1.0.0

Environment Check

环境检查

Prerequisite check: Python3 + transformers + msmodelslim required

bash

python3 --version  # Python3 >= 3.8
python3 -c "import transformers; print('OK')"  # Transformers library
python3 -c "import msmodelslim; print('OK')"  # msModelSlim library

If not installed:

pip3 install --user transformers msmodelslim

前置条件检查：需安装Python3 + transformers + msmodelslim

bash

python3 --version  # Python3 >= 3.8
python3 -c "import transformers; print('OK')"  # Transformers库
python3 -c "import msmodelslim; print('OK')"  # msModelSlim库

若未安装：

pip3 install --user transformers msmodelslim

Reference Documents

参考文档

Document	Description
Model Analysis Guide	Model structure analysis guide
Implementation Guide	Adapter implementation instructions
Registration Guide	Registration and installation guide
Verification Guide	Four-step verification workflow
Interface Checklist	Required interface implementation checklist
Core Workflow	Core workflow documentation
Acceptance Criteria	Functional acceptance criteria
Troubleshooting	Common issues and solutions

文档	描述
模型分析指南	模型结构分析指南
实现指南	适配器实现说明
注册指南	注册与安装指南
验证指南	四步验证流程
接口检查清单	必填接口实现检查清单
核心流程	核心流程文档
验收标准	功能验收标准
故障排查	常见问题与解决方案

Requirements

要求

transformers >= 4.40.0 installed
msmodelslim >= 1.0.0 installed
Transformers model to be adapted
Understanding of target quantization scheme (W8A8/W4A16)

transformers >= 4.40.0已安装
msmodelslim >= 1.0.0已安装
待适配的Transformers模型
了解目标量化方案（W8A8/W4A16）

Core Commands

核心命令

bash

undefined

bash

undefined

Create model adapter

创建模型适配器

python3 scripts/create_adapter.py
--model Qwen2-7B
--quantization W8A8

Run four-step verification

执行四步验证

python3 scripts/verify_adapter.py --adapter ./adapter.py

undefined

python3 scripts/verify_adapter.py --adapter ./adapter.py

undefined

Parameter Confirmation

参数确认

Parameter	Description	Required
model	Model name or path	Yes
quantization	Quantization scheme (W8A8/W4A16)	Yes
output	Adapter output path	No

参数	描述	是否必填
model	模型名称或路径	是
quantization	量化方案（W8A8/W4A16）	是
output	适配器输出路径	否