shunk031-transformers-convert

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

Hugging Face Transformers Model Conversion

Hugging Face Transformers模型转换

Convert custom PyTorch models to Hugging Face Transformers format while maintaining exact compatibility with the original implementation.
将自定义PyTorch模型转换为Hugging Face Transformers格式,同时保持与原始实现的完全兼容性。

Overview

概述

This skill provides a systematic workflow for transformers conversion:
  • Extract hardcoded values into PretrainedConfig
  • Create PreTrainedModel wrapper
  • Build ImageProcessor/Tokenizer
  • Test equivalence thoroughly
  • Prepare for Hub upload
Important: Use validation mode (parallel implementations) first to verify equivalence, then replace the original.
本技能提供了一套系统化的transformers转换流程:
  • 将硬编码值提取到PretrainedConfig中
  • 创建PreTrainedModel封装类
  • 构建ImageProcessor/Tokenizer
  • 全面测试等效性
  • 准备Hub上传
重要提示:先使用验证模式(并行实现)来验证等效性,再替换原始实现。

Conversion Workflow

转换流程

Step 1: Analyze the Custom Model

步骤1:分析自定义模型

Ask the user to specify:
  • Path to the custom model implementation
  • Model type (vision, text, multimodal)
  • Task (classification, segmentation, generation, etc.)
  • Validation mode or replacement mode
Then identify:
  • Model architecture and components
  • Input/output formats
  • Key hyperparameters and hardcoded values
  • Pretrained weights location
  • Preprocessing pipeline
  • Custom layers or modules
请用户指定:
  • 自定义模型实现的路径
  • 模型类型(视觉、文本、多模态)
  • 任务(分类、分割、生成等)
  • 验证模式或替换模式
然后确定:
  • 模型架构和组件
  • 输入/输出格式
  • 关键超参数和硬编码值
  • 预训练权重位置
  • 预处理流水线
  • 自定义层或模块

Step 2: Create PretrainedConfig Class

步骤2:创建PretrainedConfig类

Key principle: Extract ALL hardcoded values from the model as configurable parameters.
Template:
python
from transformers import PretrainedConfig
from typing import List, Optional

class {ModelName}Config(PretrainedConfig):
    model_type = "{model_name}"

    def __init__(
        self,
        # Core architecture parameters
        hidden_dim: int = 128,
        num_layers: int = 4,

        # Input/output parameters
        image_size: int = 1024,
        num_channels: int = 3,
        num_labels: int = 1,

        # Component-specific parameters (extract from original)
        component_param: List[int] | None = None,

        **kwargs,
    ):
        super().__init__(**kwargs)

        self.hidden_dim = hidden_dim
        self.num_layers = num_layers
        self.image_size = image_size
        self.num_channels = num_channels
        self.num_labels = num_labels

        # Use default if not specified
        self.component_param = (
            component_param if component_param is not None else [1, 2, 4]
        )
Critical: Ensure default values match what the pretrained weights expect!
核心原则:将模型中所有硬编码值提取为可配置参数。
模板:
python
from transformers import PretrainedConfig
from typing import List, Optional

class {ModelName}Config(PretrainedConfig):
    model_type = "{model_name}"

    def __init__(
        self,
        # 核心架构参数
        hidden_dim: int = 128,
        num_layers: int = 4,

        # 输入/输出参数
        image_size: int = 1024,
        num_channels: int = 3,
        num_labels: int = 1,

        # 组件特定参数(从原始模型提取)
        component_param: List[int] | None = None,

        **kwargs,
    ):
        super().__init__(**kwargs)

        self.hidden_dim = hidden_dim
        self.num_layers = num_layers
        self.image_size = image_size
        self.num_channels = num_channels
        self.num_labels = num_labels

        # 未指定时使用默认值
        self.component_param = (
            component_param if component_param is not None else [1, 2, 4]
        )
关键注意点:确保默认值与预训练权重的预期一致!

Step 3: Create PreTrainedModel Class

步骤3:创建PreTrainedModel类

Template:
python
from transformers import PreTrainedModel
from transformers.modeling_outputs import SemanticSegmenterOutput

class {ModelName}ForTask(PreTrainedModel):
    config_class = {ModelName}Config

    def __init__(self, config: {ModelName}Config):
        super().__init__(config)
        self.config = config

        # Initialize layers using config parameters (no hardcoded values!)
        self.encoder = Encoder(
            hidden_dim=config.hidden_dim,
            num_layers=config.num_layers,
        )

    def forward(
        self,
        pixel_values: torch.FloatTensor,
        labels: Optional[torch.LongTensor] = None,
        output_hidden_states: Optional[bool] = None,
        return_dict: Optional[bool] = None,
    ) -> Union[Tuple, SemanticSegmenterOutput]:
        return_dict = return_dict if return_dict is not None else self.config.use_return_dict

        # Forward pass
        logits = self.encoder(pixel_values)

        # Calculate loss if needed
        loss = None
        if labels is not None:
            # Compute loss
            pass

        if not return_dict:
            output = (logits,)
            return ((loss,) + output) if loss is not None else output

        return SemanticSegmenterOutput(
            loss=loss,
            logits=logits,
            hidden_states=None,
            attentions=None,
        )
模板:
python
from transformers import PreTrainedModel
from transformers.modeling_outputs import SemanticSegmenterOutput

class {ModelName}ForTask(PreTrainedModel):
    config_class = {ModelName}Config

    def __init__(self, config: {ModelName}Config):
        super().__init__(config)
        self.config = config

        # 使用配置参数初始化层(禁止硬编码值!)
        self.encoder = Encoder(
            hidden_dim=config.hidden_dim,
            num_layers=config.num_layers,
        )

    def forward(
        self,
        pixel_values: torch.FloatTensor,
        labels: Optional[torch.LongTensor] = None,
        output_hidden_states: Optional[bool] = None,
        return_dict: Optional[bool] = None,
    ) -> Union[Tuple, SemanticSegmenterOutput]:
        return_dict = return_dict if return_dict is not None else self.config.use_return_dict

        # 前向传播
        logits = self.encoder(pixel_values)

        # 按需计算损失
        loss = None
        if labels is not None:
            # 计算损失
            pass

        if not return_dict:
            output = (logits,)
            return ((loss,) + output) if loss is not None else output

        return SemanticSegmenterOutput(
            loss=loss,
            logits=logits,
            hidden_states=None,
            attentions=None,
        )

Step 4: Create ImageProcessor/Tokenizer

步骤4:创建ImageProcessor/Tokenizer

For vision models - Create ImageProcessor:
python
from transformers import BaseImageProcessor

class {ModelName}ImageProcessor(BaseImageProcessor):
    model_input_names = ["pixel_values"]

    def __init__(
        self,
        size: int = 1024,
        resample: str = "bilinear",
        do_normalize: bool = True,
        image_mean: List[float] | None = None,
        image_std: List[float] | None = None,
        **kwargs,
    ):
        super().__init__(**kwargs)
        self.size = size
        self.resample = resample
        self.do_normalize = do_normalize
        self.image_mean = image_mean if image_mean is not None else [0.485, 0.456, 0.406]
        self.image_std = image_std if image_std is not None else [0.229, 0.224, 0.225]

    def preprocess(
        self,
        images: ImageInput,
        return_tensors: Optional[Union[str, TensorType]] = None,
        **kwargs,
    ) -> BatchFeature:
        # Implement preprocessing matching original
        # Return BatchFeature with pixel_values
For text models - Create tokenizer configuration.
针对视觉模型 - 创建ImageProcessor:
python
from transformers import BaseImageProcessor

class {ModelName}ImageProcessor(BaseImageProcessor):
    model_input_names = ["pixel_values"]

    def __init__(
        self,
        size: int = 1024,
        resample: str = "bilinear",
        do_normalize: bool = True,
        image_mean: List[float] | None = None,
        image_std: List[float] | None = None,
        **kwargs,
    ):
        super().__init__(**kwargs)
        self.size = size
        self.resample = resample
        self.do_normalize = do_normalize
        self.image_mean = image_mean if image_mean is not None else [0.485, 0.456, 0.406]
        self.image_std = image_std if image_std is not None else [0.229, 0.224, 0.225]

    def preprocess(
        self,
        images: ImageInput,
        return_tensors: Optional[Union[str, TensorType]] = None,
        **kwargs,
    ) -> BatchFeature:
        # 实现与原始模型匹配的预处理逻辑
        # 返回包含pixel_values的BatchFeature
针对文本模型 - 创建tokenizer配置。

Step 5: Create Compatibility Tests

步骤5:创建兼容性测试

Critical: Always use the SAME preprocessed tensor for both models when comparing outputs.
python
import pytest
import torch

def test_preprocessing_matches():
    """Test that preprocessing is equivalent."""
    old_tensor = old_preprocessing(image)
    new_tensor = processor(image, return_tensors="pt")["pixel_values"][0]
    assert torch.allclose(old_tensor, new_tensor, atol=1e-6)

def test_single_image_output_matches():
    """Test that model outputs match."""
    # Load models
    old_model = OldModel()
    new_model = NewModel(NewConfig())
    new_model.load_state_dict(old_model.state_dict())

    # Prepare SAME input
    preprocessed = preprocess(image)

    with torch.no_grad():
        old_output = old_model(preprocessed)
        new_output = new_model(pixel_values=preprocessed)

    # Use 0.5% tolerance for numerical differences
    assert torch.allclose(old_output, new_output.logits, atol=5e-3, rtol=1e-2)

def test_batch_output_matches():
    """Test batch processing."""
    # Test with batch of images

def test_state_dict_compatible():
    """Test that weights can be loaded."""
    new_model.load_state_dict(old_model.state_dict())
关键注意点:比较输出时,务必对两个模型使用相同的预处理张量。
python
import pytest
import torch

def test_preprocessing_matches():
    """测试预处理逻辑是否等效。"""
    old_tensor = old_preprocessing(image)
    new_tensor = processor(image, return_tensors="pt")["pixel_values"][0]
    assert torch.allclose(old_tensor, new_tensor, atol=1e-6)

def test_single_image_output_matches():
    """测试模型输出是否匹配。"""
    # 加载模型
    old_model = OldModel()
    new_model = NewModel(NewConfig())
    new_model.load_state_dict(old_model.state_dict())

    # 准备相同的输入
    preprocessed = preprocess(image)

    with torch.no_grad():
        old_output = old_model(preprocessed)
        new_output = new_model(pixel_values=preprocessed)

    # 对数值差异使用0.5%的容差
    assert torch.allclose(old_output, new_output.logits, atol=5e-3, rtol=1e-2)

def test_batch_output_matches():
    """测试批量处理逻辑。"""
    # 使用图像批次进行测试

def test_state_dict_compatible():
    """测试权重是否可加载。"""
    new_model.load_state_dict(old_model.state_dict())

Step 6: Create MODEL_CARD.md

步骤6:创建MODEL_CARD.md

Generate comprehensive model card following Hugging Face standards. Include:
  • Model description
  • Usage examples
  • Training details
  • Evaluation metrics
  • Citation information
See Hugging Face model card documentation for template.
遵循Hugging Face标准生成完整的模型卡片,包含:
  • 模型描述
  • 使用示例
  • 训练细节
  • 评估指标
  • 引用信息
请参考Hugging Face模型卡片文档获取模板。

Step 7: Create Hub Push Script

步骤7:创建Hub推送脚本

Critical: Register classes with
register_for_auto_class()
before pushing.
python
#!/usr/bin/env python3
from huggingface_hub import HfApi
from {module}.transformers import {ModelName}Config, {ModelName}ForTask, {ModelName}ImageProcessor

def main():
    # CRITICAL: Register for Auto* support
    {ModelName}Config.register_for_auto_class()
    {ModelName}ForTask.register_for_auto_class("AutoModel")
    {ModelName}ImageProcessor.register_for_auto_class("AutoImageProcessor")

    # Load original model
    original_model = OriginalModel()

    # Create transformers-compatible model
    config = {ModelName}Config()
    model = {ModelName}ForTask(config)
    model.load_state_dict(original_model.state_dict())

    # Save and push
    model.save_pretrained(local_dir)
    config.save_pretrained(local_dir)
    processor = {ModelName}ImageProcessor()
    processor.save_pretrained(local_dir)

    api = HfApi(token=token)
    api.create_repo(repo_id=repo_id, exist_ok=True)
    api.upload_folder(repo_id=repo_id, folder_path=local_dir)
关键注意点:推送前需使用
register_for_auto_class()
注册类。
python
#!/usr/bin/env python3
from huggingface_hub import HfApi
from {module}.transformers import {ModelName}Config, {ModelName}ForTask, {ModelName}ImageProcessor

def main():
    # 关键步骤:注册以支持Auto*类
    {ModelName}Config.register_for_auto_class()
    {ModelName}ForTask.register_for_auto_class("AutoModel")
    {ModelName}ImageProcessor.register_for_auto_class("AutoImageProcessor")

    # 加载原始模型
    original_model = OriginalModel()

    # 创建兼容transformers的模型
    config = {ModelName}Config()
    model = {ModelName}ForTask(config)
    model.load_state_dict(original_model.state_dict())

    # 保存并推送
    model.save_pretrained(local_dir)
    config.save_pretrained(local_dir)
    processor = {ModelName}ImageProcessor()
    processor.save_pretrained(local_dir)

    api = HfApi(token=token)
    api.create_repo(repo_id=repo_id, exist_ok=True)
    api.upload_folder(repo_id=repo_id, folder_path=local_dir)

Implementation Strategy

实现策略

Validation Mode (Recommended First)

验证模式(推荐优先使用)

Create parallel implementations:
{project}/
├── {module}/
│   ├── original_model.py           # Existing
│   └── transformers/               # NEW - for validation
│       ├── __init__.py
│       ├── configuration_{model}.py
│       ├── modeling_{model}.py
│       └── processing_{model}.py
└── tests/
    └── test_transformers_compatibility.py
Workflow:
  1. Create transformers/ package alongside original
  2. Run compatibility tests
  3. Debug any discrepancies
  4. Once tests pass → proceed to replacement
创建并行实现:
{project}/
├── {module}/
│   ├── original_model.py           # 现有文件
│   └── transformers/               # 新增目录 - 用于验证
│       ├── __init__.py
│       ├── configuration_{model}.py
│       ├── modeling_{model}.py
│       └── processing_{model}.py
└── tests/
    └── test_transformers_compatibility.py
流程:
  1. 在原始模型旁创建transformers/包
  2. 运行兼容性测试
  3. 调试所有差异
  4. 测试通过后 → 进入替换阶段

Replacement Mode (After Validation)

替换模式(验证完成后)

Once equivalence is verified:
{project}/
└── {module}/
    ├── configuration_{model}.py    # Replaces original
    ├── modeling_{model}.py
    └── processing_{model}.py
Workflow:
  1. Remove or archive original implementation
  2. Move transformers/* files up one level
  3. Update all imports
  4. Update tests
  5. Update documentation
验证等效性后:
{project}/
└── {module}/
    ├── configuration_{model}.py    # 替换原始实现
    ├── modeling_{model}.py
    └── processing_{model}.py
流程:
  1. 删除或归档原始实现
  2. 将transformers/下的文件移到上一级目录
  3. 更新所有导入语句
  4. 更新测试
  5. 更新文档

Common Issues

常见问题

For detailed troubleshooting, see references/common-pitfalls.md.
Quick reference:
  • Hardcoded values: Extract to config with matching defaults
  • Preprocessing mismatches: Use exact same pipeline and parameters
  • State dict keys: Keep layer names matching original
  • Test tolerance: Use 0.5% tolerance (5e-3) for numerical differences
  • Device handling: Use
    self.device
    from PreTrainedModel
  • Image size order:
    post_process_semantic_segmentation()
    expects
    (width, height)
如需详细故障排除,请参考references/common-pitfalls.md
快速参考:
  • 硬编码值:提取到配置中并设置匹配的默认值
  • 预处理不匹配:使用完全相同的流水线和参数
  • 状态字典键:保持层名称与原始模型一致
  • 测试容差:对数值差异使用0.5%的容差(5e-3)
  • 设备处理:使用PreTrainedModel中的
    self.device
  • 图像尺寸顺序
    post_process_semantic_segmentation()
    期望格式为
    (width, height)

Debugging Strategy

调试策略

If outputs don't match:
  1. Check preprocessing produces identical tensors
  2. Check state dict loaded correctly
  3. Step-by-step comparison of each layer
  4. Check device placement
  5. Check determinism with torch.manual_seed
See references/common-pitfalls.md for detailed debugging steps.
如果输出不匹配:
  1. 检查预处理是否生成相同的张量
  2. 检查状态字典是否正确加载
  3. 逐层对比输出
  4. 检查设备放置
  5. 使用torch.manual_seed确保确定性
详细调试步骤请参考references/common-pitfalls.md

Project Learnings

项目经验总结

After completing a conversion, add learnings to references/learnings.md.
This accumulates knowledge from each project to avoid repeating mistakes.
完成转换后,请将经验添加到references/learnings.md中。
这将积累每个项目的知识,避免重复犯错。

Checklist

检查清单

  • Step 1: Model analyzed
    • Architecture identified
    • Hyperparameters extracted
    • Preprocessing understood
  • Step 2: Config class created
    • All hardcoded values moved to config
    • Defaults match pretrained weights
    • Type hints added
  • Step 3: Model class created
    • Inherits from PreTrainedModel
    • Uses only config parameters
    • Forward signature matches conventions
  • Step 4: Processor created
    • Preprocessing matches original
    • Returns appropriate format
  • Step 5: Tests created
    • Preprocessing test passes
    • Single image test passes
    • Batch test passes
    • State dict test passes
  • Step 6: MODEL_CARD.md created
    • All sections filled
    • Usage examples tested
    • Citations included
  • Step 7: Push script created
    • register_for_auto_class() called
    • Script tested locally
    • Successfully pushed to Hub
  • Cleanup (if doing replacement)
    • Original implementation removed/archived
    • Imports updated
    • Documentation updated
  • 步骤1:模型已分析
    • 架构已确定
    • 超参数已提取
    • 预处理逻辑已理解
  • 步骤2:Config类已创建
    • 所有硬编码值已移至配置
    • 默认值与预训练权重匹配
    • 添加了类型提示
  • 步骤3:Model类已创建
    • 继承自PreTrainedModel
    • 仅使用配置参数
    • 前向传播签名符合规范
  • 步骤4:Processor已创建
    • 预处理逻辑与原始模型匹配
    • 返回合适的格式
  • 步骤5:测试已创建
    • 预处理测试通过
    • 单图像测试通过
    • 批量测试通过
    • 状态字典测试通过
  • 步骤6:MODEL_CARD.md已创建
    • 所有部分已填写
    • 使用示例已测试
    • 包含引用信息
  • 步骤7:推送脚本已创建
    • 已调用register_for_auto_class()
    • 脚本已本地测试
    • 已成功推送到Hub
  • 清理(如果使用替换模式)
    • 原始实现已删除/归档
    • 导入语句已更新
    • 文档已更新