Complete Guide to Pipeline Variable Management

Skill Overview

Skill Name: Pipeline Variable Management
Applicable Scenarios: Full lifecycle management and field extension of pipeline variables
Importance: ⭐⭐⭐⭐ (High Priority)
Document Version: 2.0
Last Updated: 2025-01

Pipeline variables are a core feature in BK-CI pipelines, supporting dynamic transfer and management of data during pipeline orchestration and build execution. This Skill provides two core themes for variable management:

Variable Lifecycle - Complete process of variables from creation to destruction
Variable Field Extension - How to extend the data structure of variables

I. Overview of Variable Management Architecture

1.1 Definition and Role of Variables

Pipeline variables are the core mechanism for storing and transferring data in pipelines, supporting:

✅ Configuration Parameterization - Replace fixed values in pipeline configurations with variables
✅ Data Transfer - Transfer data between different Stage/Job/Task
✅ Dynamic Calculation - Dynamically generate and update variable values at runtime
✅ User Input - Users input parameter values when triggering manually
✅ System Built-in - Provide system-level predefined variables

1.2 Two Core Models of Variables

There is bidirectional conversion between two data models for pipeline variables in BK-CI:

1. BuildFormProperty (Backend Internal Model)

Location:

common-pipeline/src/main/kotlin/.../BuildFormProperty.kt

Purpose: Used internally in microservices, database storage format
Features: Complete field definition, including Swagger annotations
Application: Internal processing of Process module, data persistence

2. Variable (YAML Model)

Location:

common-pipeline-yaml/src/main/kotlin/.../yaml/v3/models/Variable.kt

Purpose: YAML pipeline definition, external API interaction
Features: Jackson annotations, supports JSON serialization
Application: PAC pipelines, OpenAPI interfaces

3. Converter (VariableTransfer)

Location:

common-pipeline-yaml/src/main/kotlin/.../yaml/transfer/VariableTransfer.kt

Responsibilities: Implements bidirectional conversion between
```
BuildFormProperty
```
↔
```
Variable
```
Core Methods:
- ```
transfer(BuildFormProperty): Variable
```
  - Internal model → YAML model
- ```
transfer(Variable): BuildFormProperty
```
  - YAML model → Internal model

1.3 Storage and Transfer of Variables

┌─────────────────────────────────────────────────────────────────┐
│                       Variable Storage and Transfer Architecture                          │
└─────────────────────────────────────────────────────────────────┘

  ┌──────────────────┐
  │  Pipeline Configuration Storage   │  BuildFormProperty
  │  (T_PIPELINE_*) │  Stored in Model JSON
  └────────┬─────────┘
           │
           ▼
  ┌──────────────────┐
  │   Build Start       │  Read variable definitions
  │  (StartBuild)   │  Initialize variable values
  └────────┬─────────┘
           │
           ▼
  ┌──────────────────┐
  │  Runtime Variable Storage   │  T_PIPELINE_BUILD_VAR
  │  (BuildVar)     │  Build-level variable instances
  └────────┬─────────┘
           │
           ├─────────────────┬─────────────────┐
           ▼                 ▼                 ▼
  ┌────────────────┐ ┌────────────────┐ ┌────────────────┐
  │   Stage 1      │ │   Stage 2      │ │   Stage 3      │
  │  Inherit + Add   │ │  Inherit + Update   │ │  Inherit + Output   │
  └────────────────┘ └────────────────┘ └────────────────┘

II. Two Core Themes

2.1 Variable Lifecycle Management

Covered Content:

🔄 Variable Creation and Initialization
📦 Variable Storage Mechanism
🔀 Variable Transfer and Inheritance
🔄 Dynamic Variable Update
🔍 Variable Query and Debugging

Typical Scenarios:

Understand the complete flow of variables during build execution
Develop new variable initialization logic
Debug variable transfer issues
Implement dynamic variable update functions

Refer to Documentation: reference/1-lifecycle.md (1414 lines)

2.2 Variable Field Extension

Covered Content:

🆕 Add new variable fields
🔧 Field type definition
🔄 Model conversion processing
🎯 Frontend and backend field synchronization

Typical Scenarios:

Add new attribute fields to variables
Extend variable types
Handle historical data compatibility
Synchronize frontend and backend field definitions

Refer to Documentation: reference/2-extension.md (534 lines)

III. Usage Guide

3.1 Scenario 1: Understand Variable Lifecycle

Example Questions:

"How are variable values transferred from pipeline configurations to build execution?"
"How do plugins obtain and update variable values?"
"How is the scope of variables controlled?"

Action: Refer to reference/1-lifecycle.md

Content Includes:

Panoramic view of variable lifecycle
Detailed explanation of six key stages
Variable storage table structure
Variable transfer mechanism
Debugging methods and common issues

3.2 Scenario 2: Extend Variable Fields

Example Questions:

"How to add a new
```
allowModify
```
field to variables?"
"How to extend the optional value list of variables?"
"How to ensure correct conversion of new fields between YAML and internal models?"

Action: Refer to reference/2-extension.md

Content Includes:

Complete path for variable field extension
BuildFormProperty field definition
Variable (YAML) field definition
VariableTransfer converter implementation
Frontend field synchronization
Testing and verification

IV. Quick Reference for Core Classes and Files

4.1 Variable Definition Classes

Class Name	Location	Purpose
`BuildFormProperty`	`common-pipeline/.../BuildFormProperty.kt`	Backend internal variable model
`Variable`	`common-pipeline-yaml/.../Variable.kt`	YAML variable model
`VariableTransfer`	`common-pipeline-yaml/.../VariableTransfer.kt`	Model converter

4.2 Variable Processing Services

Class Name Location Responsibility

Class Name	Location	Responsibility
`PipelineVariableService`	`process/biz-base/.../PipelineVariableService.kt`	Variable business logic
`BuildVariableService`	`process/biz-base/.../BuildVariableService.kt`	Build variable management
`VariableAcrossInfoUtil`	`process/biz-base/.../VariableAcrossInfoUtil.kt`	Cross-Job variable transfer

PipelineVariableService

process/biz-base/.../PipelineVariableService.kt

Variable business logic

BuildVariableService

process/biz-base/.../BuildVariableService.kt

Build variable management

VariableAcrossInfoUtil

process/biz-base/.../VariableAcrossInfoUtil.kt

Cross-Job variable transfer

4.3 Database Tables

Table Name	Purpose
`T_PIPELINE_BUILD_VAR`	Build-level variable instances (runtime)
`T_PIPELINE_SETTING`	Pipeline configuration (includes variable definitions)
`T_PIPELINE_MODEL_TASK`	Plugin configuration (references variables)

V. Development Process

5.1 Variable Function Development Process

1. Clarify Requirements
   ↓
2. Determine the involved lifecycle stages
   │  → Refer to reference/1-lifecycle.md
   ↓
3. Need to extend fields?
   ├─ Yes → Refer to reference/2-extension.md
   │       Execute field extension process
   ↓
4. Implement Backend Logic
   │  → Modify Service/DAO layers
   ↓
5. Frontend Synchronization (if needed)
   │  → Update Vue components
   ↓
6. Testing and Verification
   │  → Unit tests + Integration tests
   ↓
7. Document Update

5.2 Common Development Tasks

Task	Reference Document	Key Classes
Add new variable types	2-extension.md	`BuildFormProperty` , `Variable`
Variable initialization logic	1-lifecycle.md	`BuildVariableService`
Cross-Job transfer	1-lifecycle.md	`VariableAcrossInfoUtil`
Variable permission control	1-lifecycle.md	`PipelineVariableService`
Variable expression parsing	-	`utility-components` Skill (reference/2-expression-parser.md)

VI. Relationship with Other Skills

6.1 Dependent Skills

pipeline-model-architecture
- Understand the position of variables in the Model
process-module-architecture
- Understand the code architecture of variable processing
utility-components
(reference/2-expression-parser.md) - Expression parsing for variable references

6.2 Referenced Scenarios

PAC Pipelines → Definition of variables in YAML
Pipeline Orchestration → Creation and configuration of variables
Build Execution → Initialization and transfer of variables
Plugin Development → How plugins read and update variables

VII. Detailed Document Navigation

Document	Content	Lines	File Size
1-lifecycle.md	Complete variable lifecycle process	1414	78.7KB
2-extension.md	Variable field extension guide	534	20.2KB

Usage Suggestions:

Beginners: First read this document to understand the overall architecture, then dive into 1-lifecycle.md

Developers: Directly refer to the corresponding reference document according to specific development tasks

Debugging: Prioritize checking the debugging section in 1-lifecycle.md

VIII. FAQ

Q1: What's the difference between variables and parameters?

A: In BK-CI, "variables" and "parameters" are essentially the same concept, both referring to

BuildFormProperty

. The difference lies in usage scenarios:

Parameters: Variables input by users when starting a pipeline
Variables: Variables defined and used internally in the pipeline

Q2: How to implement cross-Stage variable transfer?

A: Refer to the "Stage 4: Cross-container Transfer" section in reference/1-lifecycle.md to learn how to use

VariableAcrossInfoUtil

Q3: What to do if the frontend doesn't display after adding a new field?

A: Follow the "Step 4: Frontend Field Synchronization" section in reference/2-extension.md to ensure that the frontend Vue components have been updated synchronously.

Q4: Variable values are empty or incorrect?

A: Refer to the "Debugging Methods" section in reference/1-lifecycle.md and use logs and databases to troubleshoot.

IX. Version History

Version	Date	Update Content
2.0	2025-01	Integrate `pipeline-variable-lifecycle` and `pipeline-variable-extension` , adopt reference structure
1.x	2024-12	Two independent Skills

pipeline-variable-management

NPX Install

Tags

SKILL.md Content (Chinese)

Complete Guide to Pipeline Variable Management

Skill Overview

I. Overview of Variable Management Architecture

1.1 Definition and Role of Variables

1.2 Two Core Models of Variables

1. BuildFormProperty (Backend Internal Model)

2. Variable (YAML Model)

3. Converter (VariableTransfer)

1.3 Storage and Transfer of Variables

II. Two Core Themes

2.1 Variable Lifecycle Management

2.2 Variable Field Extension

III. Usage Guide

3.1 Scenario 1: Understand Variable Lifecycle

3.2 Scenario 2: Extend Variable Fields

IV. Quick Reference for Core Classes and Files

4.1 Variable Definition Classes

4.2 Variable Processing Services

4.3 Database Tables

V. Development Process

5.1 Variable Function Development Process

5.2 Common Development Tasks

VI. Relationship with Other Skills

6.1 Dependent Skills

6.2 Referenced Scenarios

VII. Detailed Document Navigation

VIII. FAQ

Q1: What's the difference between variables and parameters?

Q2: How to implement cross-Stage variable transfer?

Q3: What to do if the frontend doesn't display after adding a new field?

Q4: Variable values are empty or incorrect?

IX. Version History