ci-cd
Compare original and translation side by side
🇺🇸
Original
English🇨🇳
Translation
ChineseCI/CD
CI/CD
Overview
概述
This skill covers the complete lifecycle of CI/CD pipeline design, implementation, and optimization across platforms including GitHub Actions, GitLab CI, Jenkins, CircleCI, and cloud-native solutions. It encompasses automated testing integration, security scanning, artifact management, deployment strategies, and specialized pipelines for ML workloads.
本技能涵盖跨平台CI/CD流水线的设计、实现与优化全生命周期，涉及平台包括GitHub Actions、GitLab CI、Jenkins、CircleCI以及云原生解决方案。内容包含自动化测试集成、安全扫描、制品管理、部署策略，以及针对机器学习工作负载的专用流水线。
When to Use
适用场景
- Implementing or migrating CI/CD pipelines
- Optimizing build and test execution times
- Integrating security scanning (SAST, DAST, dependency checks)
- Setting up deployment automation with rollback strategies
- Configuring test suites in CI environments
- Managing artifacts and container registries
- Implementing ML model training and deployment pipelines
- Troubleshooting pipeline failures and flakiness
- 实现或迁移CI/CD流水线
- 优化构建与测试执行时间
- 集成安全扫描（SAST、DAST、依赖检查）
- 配置带有回滚策略的部署自动化
- 在CI环境中配置测试套件
- 管理制品与容器镜像仓库
- 实现机器学习模型训练与部署流水线
- 排查流水线失败与不稳定问题
Instructions
操作步骤
1. Analyze Requirements
1. 需求分析
- Identify build and test requirements
- Determine deployment targets and environments
- Assess security scanning needs (SAST, DAST, secrets, dependencies)
- Plan environment promotion strategy (dev → staging → production)
- Define quality gates and approval workflows
- Identify test suite composition (unit, integration, E2E)
- Determine artifact storage and retention policies
- 明确构建与测试需求
- 确定部署目标与环境
- 评估安全扫描需求（SAST、DAST、密钥检测、依赖检查）
- 规划环境升级策略（开发→预发布→生产）
- 定义质量门禁与审批流程
- 确定测试套件组成（单元测试、集成测试、端到端测试）
- 制定制品存储与保留策略
2. Design Pipeline Architecture
2. 流水线架构设计
- Structure stages logically with clear dependencies
- Optimize for speed through parallelization and caching
- Design fail-fast strategy (lint → unit tests → integration tests → build)
- Plan secret management and secure credential handling
- Define deployment strategies (rolling, blue-green, canary)
- Architect for rollback and recovery procedures
- Design matrix builds for multi-platform support
- Plan monorepo CI strategies if applicable
- 按逻辑划分阶段并明确依赖关系
- 通过并行化与缓存优化执行速度
- 设计快速失败策略（代码检查→单元测试→集成测试→构建）
- 规划密钥管理与安全凭证处理方案
- 定义部署策略（滚动发布、蓝绿发布、金丝雀发布）
- 设计回滚与恢复流程
- 配置多平台支持的矩阵构建
- 若适用，规划单体仓库CI策略
3. Implement Testing Integration
3. 测试集成实现
- Configure unit test execution with coverage reporting
- Set up integration tests with service dependencies (databases, APIs)
- Implement E2E/smoke tests for critical user journeys
- Configure test parallelization and sharding
- Integrate test result reporting (JUnit, TAP, JSON)
- Set up flaky test detection and quarantine
- Configure performance/load testing stages
- Implement visual regression testing if applicable
- 配置单元测试执行与覆盖率报告
- 搭建带有服务依赖（数据库、API）的集成测试环境
- 实现针对关键用户路径的端到端/冒烟测试
- 配置测试并行化与分片
- 集成测试结果报告（JUnit、TAP、JSON格式）
- 配置不稳定测试检测与隔离机制
- 配置性能/负载测试阶段
- 若适用，实现视觉回归测试
4. Implement Security Scanning
4. 安全扫描实现
- Integrate SAST (static analysis) tools (SonarQube, CodeQL, Semgrep)
- Configure DAST (dynamic analysis) for deployed environments
- Set up dependency/vulnerability scanning (Dependabot, Snyk, Trivy)
- Implement container image scanning
- Configure secrets detection (GitGuardian, TruffleHog)
- Set up license compliance checking
- Define security gate thresholds and failure policies
- 集成SAST（静态代码分析）工具（SonarQube、CodeQL、Semgrep）
- 为已部署环境配置DAST（动态代码分析）
- 搭建依赖/漏洞扫描（Dependabot、Snyk、Trivy）
- 实现容器镜像扫描
- 配置密钥检测（GitGuardian、TruffleHog）
- 搭建许可证合规性检查
- 定义安全门禁阈值与失败策略
5. Implement Build and Artifact Management
5. 构建与制品管理实现
- Configure dependency caching strategies
- Implement build output caching and layer caching (Docker)
- Set up artifact versioning and tagging
- Configure container registry integration
- Implement multi-stage builds for optimization
- Set up artifact signing and attestation
- Configure artifact retention and cleanup policies
- 配置依赖缓存策略
- 实现构建输出缓存与分层缓存（Docker）
- 搭建制品版本控制与标签管理
- 配置容器镜像仓库集成
- 实现多阶段构建以优化性能
- 搭建制品签名与验证
- 配置制品保留与清理策略
6. Implement Deployment Automation
6. 部署自动化实现
- Configure environment-specific deployments
- Implement deployment strategies (rolling, blue-green, canary)
- Set up health checks and readiness probes
- Configure smoke tests post-deployment
- Implement automated rollback on failure
- Set up deployment notifications (Slack, email, PagerDuty)
- Configure manual approval gates for production
- 配置环境专属部署流程
- 实现部署策略（滚动发布、蓝绿发布、金丝雀发布）
- 搭建健康检查与就绪探针
- 配置部署后冒烟测试
- 实现失败时自动回滚
- 搭建部署通知（Slack、邮件、PagerDuty）
- 配置生产环境手动审批门禁
7. Optimize Pipeline Performance
7. 流水线性能优化
- Analyze pipeline execution times and bottlenecks
- Implement job parallelization for independent tasks
- Configure aggressive caching (dependencies, build outputs, Docker layers)
- Optimize test execution (parallel runners, test sharding)
- Use matrix builds efficiently
- Consider self-hosted runners for performance-critical workloads
- Implement conditional job execution (path filters, change detection)
- 分析流水线执行时间与瓶颈
- 为独立任务实现作业并行化
- 配置激进缓存（依赖、构建输出、Docker分层）
- 优化测试执行（并行运行器、测试分片）
- 高效使用矩阵构建
- 考虑使用自托管运行器处理性能关键型工作负载
- 实现条件作业执行（路径过滤、变更检测）
8. Ensure Reliability and Observability
8. 可靠性与可观测性保障
- Add retry logic for transient failures
- Implement comprehensive error handling
- Configure alerts for pipeline failures
- Set up metrics and dashboards for pipeline health
- Document runbooks and troubleshooting procedures
- Implement audit logging for deployments
- Configure SLO tracking for pipeline performance
- 为临时失败添加重试逻辑
- 实现全面的错误处理
- 配置流水线失败告警
- 搭建流水线健康指标与仪表盘
- 编写运行手册与排查流程
- 实现部署审计日志
- 配置流水线性能SLO跟踪
Best Practices
最佳实践
Core Principles
核心原则
- Fail Fast: Run cheap, fast checks first (lint, type check, unit tests)
- Parallelize Aggressively: Run independent jobs concurrently
- Cache Everything: Dependencies, build outputs, Docker layers
- Secure by Default: Secrets in vaults, least privilege, audit logs
- Environment Parity: Keep dev/staging/prod as similar as possible
- Immutable Artifacts: Build once, promote everywhere
- Automated Rollback: Every deployment must be reversible
- Idempotent Operations: Pipelines should be safely re-runnable
- 快速失败：优先执行低成本、快速的检查（代码规范、类型检查、单元测试）
- 激进并行化：并发执行独立作业
- 全面缓存：缓存依赖、构建输出、Docker分层
- 默认安全：密钥存储在保险箱中，遵循最小权限原则，保留审计日志
- 环境一致性：保持开发/预发布/生产环境尽可能相似
- 不可变制品：一次构建，多处部署
- 自动回滚：每个部署都必须可回滚
- 幂等操作：流水线可安全重复运行
Testing in CI/CD
CI/CD中的测试
- Test Pyramid: More unit tests, fewer integration tests, minimal E2E
- Isolation: Tests should not depend on execution order
- Determinism: Eliminate flaky tests or quarantine them
- Fast Feedback: Unit tests < 5min, full suite < 15min target
- Coverage Gates: Enforce minimum coverage thresholds
- Service Mocking: Use test doubles for external dependencies
- 测试金字塔：更多单元测试，更少集成测试，最少端到端测试
- 隔离性：测试不应依赖执行顺序
- 确定性：消除不稳定测试或隔离它们
- 快速反馈：单元测试<5分钟，全套件目标<15分钟
- 覆盖率门禁：强制执行最低覆盖率阈值
- 服务模拟：对外部依赖使用测试替身
Security
安全
- Shift Left: Run security scans early in the pipeline
- Dependency Scanning: Check for CVEs in all dependencies
- Secrets Management: Never hardcode secrets, use secure vaults
- Least Privilege: Minimal permissions for pipeline runners
- Supply Chain Security: Verify and sign artifacts
- Audit Trail: Log all deployments and access
- 左移安全：在流水线早期运行安全扫描
- 依赖扫描：检查所有依赖中的CVE漏洞
- 密钥管理：绝不硬编码密钥，使用安全保险箱
- 最小权限：为流水线运行器分配最小权限
- 供应链安全：验证并签名制品
- 审计追踪：记录所有部署与访问操作
Performance
性能
- Incremental Builds: Only rebuild changed components
- Layer Caching: Optimize Dockerfile layer order
- Dependency Locking: Pin versions for reproducibility
- Resource Limits: Prevent resource exhaustion
- Path Filtering: Skip jobs when irrelevant files change
- 增量构建：仅重新构建变更的组件
- 分层缓存：优化Dockerfile分层顺序
- 依赖锁定：固定版本以确保可复现性
- 资源限制：防止资源耗尽
- 路径过滤：无关文件变更时跳过作业
Examples
示例
Example 1: GitHub Actions Workflow
示例1：GitHub Actions 工作流
yaml
name: CI/CD Pipeline
on:
push:
branches: [main, develop]
pull_request:
branches: [main]
env:
NODE_VERSION: "20"
REGISTRY: ghcr.io
IMAGE_NAME: ${{ github.repository }}
jobs:
lint:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: ${{ env.NODE_VERSION }}
cache: "npm"
- name: Install dependencies
run: npm ci
- name: Run linter
run: npm run lint
test:
runs-on: ubuntu-latest
needs: lint
services:
postgres:
image: postgres:16
env:
POSTGRES_PASSWORD: postgres
POSTGRES_DB: test
ports:
- 5432:5432
options: >-
--health-cmd pg_isready
--health-interval 10s
--health-timeout 5s
--health-retries 5
steps:
- uses: actions/checkout@v4
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: ${{ env.NODE_VERSION }}
cache: "npm"
- name: Install dependencies
run: npm ci
- name: Run tests
run: npm test -- --coverage
env:
DATABASE_URL: postgresql://postgres:postgres@localhost:5432/test
- name: Upload coverage
uses: codecov/codecov-action@v3
with:
files: ./coverage/lcov.info
build:
runs-on: ubuntu-latest
needs: test
permissions:
contents: read
packages: write
steps:
- uses: actions/checkout@v4
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Log in to Container Registry
uses: docker/login-action@v3
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Extract metadata
id: meta
uses: docker/metadata-action@v5
with:
images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
tags: |
type=ref,event=branch
type=ref,event=pr
type=sha,prefix=
type=raw,value=latest,enable=${{ github.ref == 'refs/heads/main' }}
- name: Build and push
uses: docker/build-push-action@v5
with:
context: .
push: true
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
cache-from: type=gha
cache-to: type=gha,mode=max
deploy-staging:
runs-on: ubuntu-latest
needs: build
if: github.ref == 'refs/heads/develop'
environment: staging
steps:
- uses: actions/checkout@v4
- name: Deploy to staging
uses: azure/k8s-deploy@v4
with:
namespace: staging
manifests: |
k8s/deployment.yaml
k8s/service.yaml
images: |
${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ github.sha }}
deploy-production:
runs-on: ubuntu-latest
needs: build
if: github.ref == 'refs/heads/main'
environment: production
steps:
- uses: actions/checkout@v4
- name: Deploy to production
uses: azure/k8s-deploy@v4
with:
namespace: production
manifests: |
k8s/deployment.yaml
k8s/service.yaml
images: |
${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ github.sha }}
strategy: canary
percentage: 20yaml
name: CI/CD Pipeline
on:
push:
branches: [main, develop]
pull_request:
branches: [main]
env:
NODE_VERSION: "20"
REGISTRY: ghcr.io
IMAGE_NAME: ${{ github.repository }}
jobs:
lint:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: ${{ env.NODE_VERSION }}
cache: "npm"
- name: Install dependencies
run: npm ci
- name: Run linter
run: npm run lint
test:
runs-on: ubuntu-latest
needs: lint
services:
postgres:
image: postgres:16
env:
POSTGRES_PASSWORD: postgres
POSTGRES_DB: test
ports:
- 5432:5432
options: >-
--health-cmd pg_isready
--health-interval 10s
--health-timeout 5s
--health-retries 5
steps:
- uses: actions/checkout@v4
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: ${{ env.NODE_VERSION }}
cache: "npm"
- name: Install dependencies
run: npm ci
- name: Run tests
run: npm test -- --coverage
env:
DATABASE_URL: postgresql://postgres:postgres@localhost:5432/test
- name: Upload coverage
uses: codecov/codecov-action@v3
with:
files: ./coverage/lcov.info
build:
runs-on: ubuntu-latest
needs: test
permissions:
contents: read
packages: write
steps:
- uses: actions/checkout@v4
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Log in to Container Registry
uses: docker/login-action@v3
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Extract metadata
id: meta
uses: docker/metadata-action@v5
with:
images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
tags: |
type=ref,event=branch
type=ref,event=pr
type=sha,prefix=
type=raw,value=latest,enable=${{ github.ref == 'refs/heads/main' }}
- name: Build and push
uses: docker/build-push-action@v5
with:
context: .
push: true
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
cache-from: type=gha
cache-to: type=gha,mode=max
deploy-staging:
runs-on: ubuntu-latest
needs: build
if: github.ref == 'refs/heads/develop'
environment: staging
steps:
- uses: actions/checkout@v4
- name: Deploy to staging
uses: azure/k8s-deploy@v4
with:
namespace: staging
manifests: |
k8s/deployment.yaml
k8s/service.yaml
images: |
${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ github.sha }}
deploy-production:
runs-on: ubuntu-latest
needs: build
if: github.ref == 'refs/heads/main'
environment: production
steps:
- uses: actions/checkout@v4
- name: Deploy to production
uses: azure/k8s-deploy@v4
with:
namespace: production
manifests: |
k8s/deployment.yaml
k8s/service.yaml
images: |
${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ github.sha }}
strategy: canary
percentage: 20Example 2: GitLab CI Pipeline
示例2：GitLab CI 流水线
yaml
stages:
- validate
- test
- build
- deploy
variables:
DOCKER_TLS_CERTDIR: "/certs"
IMAGE_TAG: $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
.node-base:
image: node:20-alpine
cache:
key: ${CI_COMMIT_REF_SLUG}
paths:
- node_modules/
lint:
stage: validate
extends: .node-base
script:
- npm ci
- npm run lint
rules:
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
- if: $CI_COMMIT_BRANCH == "main"
test:
stage: test
extends: .node-base
services:
- postgres:16
variables:
POSTGRES_DB: test
POSTGRES_USER: runner
POSTGRES_PASSWORD: runner
DATABASE_URL: postgresql://runner:runner@postgres:5432/test
script:
- npm ci
- npm test -- --coverage
coverage: '/Lines\s*:\s*(\d+\.?\d*)%/'
artifacts:
reports:
coverage_report:
coverage_format: cobertura
path: coverage/cobertura-coverage.xml
junit: junit.xml
build:
stage: build
image: docker:24
services:
- docker:24-dind
script:
- docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
- docker build -t $IMAGE_TAG .
- docker push $IMAGE_TAG
rules:
- if: $CI_COMMIT_BRANCH == "main"
- if: $CI_COMMIT_BRANCH == "develop"
deploy-staging:
stage: deploy
image: bitnami/kubectl:latest
script:
- kubectl set image deployment/app app=$IMAGE_TAG -n staging
- kubectl rollout status deployment/app -n staging --timeout=300s
environment:
name: staging
url: https://staging.example.com
rules:
- if: $CI_COMMIT_BRANCH == "develop"
deploy-production:
stage: deploy
image: bitnami/kubectl:latest
script:
- kubectl set image deployment/app app=$IMAGE_TAG -n production
- kubectl rollout status deployment/app -n production --timeout=300s
environment:
name: production
url: https://example.com
when: manual
rules:
- if: $CI_COMMIT_BRANCH == "main"yaml
stages:
- validate
- test
- build
- deploy
variables:
DOCKER_TLS_CERTDIR: "/certs"
IMAGE_TAG: $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
.node-base:
image: node:20-alpine
cache:
key: ${CI_COMMIT_REF_SLUG}
paths:
- node_modules/
lint:
stage: validate
extends: .node-base
script:
- npm ci
- npm run lint
rules:
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
- if: $CI_COMMIT_BRANCH == "main"
test:
stage: test
extends: .node-base
services:
- postgres:16
variables:
POSTGRES_DB: test
POSTGRES_USER: runner
POSTGRES_PASSWORD: runner
DATABASE_URL: postgresql://runner:runner@postgres:5432/test
script:
- npm ci
- npm test -- --coverage
coverage: '/Lines\s*:\s*(\d+\.?\d*)%/'
artifacts:
reports:
coverage_report:
coverage_format: cobertura
path: coverage/cobertura-coverage.xml
junit: junit.xml
build:
stage: build
image: docker:24
services:
- docker:24-dind
script:
- docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
- docker build -t $IMAGE_TAG .
- docker push $IMAGE_TAG
rules:
- if: $CI_COMMIT_BRANCH == "main"
- if: $CI_COMMIT_BRANCH == "develop"
deploy-staging:
stage: deploy
image: bitnami/kubectl:latest
script:
- kubectl set image deployment/app app=$IMAGE_TAG -n staging
- kubectl rollout status deployment/app -n staging --timeout=300s
environment:
name: staging
url: https://staging.example.com
rules:
- if: $CI_COMMIT_BRANCH == "develop"
deploy-production:
stage: deploy
image: bitnami/kubectl:latest
script:
- kubectl set image deployment/app app=$IMAGE_TAG -n production
- kubectl rollout status deployment/app -n production --timeout=300s
environment:
name: production
url: https://example.com
when: manual
rules:
- if: $CI_COMMIT_BRANCH == "main"Example 3: Reusable Workflow (GitHub Actions)
示例3：可复用工作流（GitHub Actions）
yaml
undefinedyaml
undefined.github/workflows/reusable-deploy.yml
.github/workflows/reusable-deploy.yml
name: Reusable Deploy Workflow
on:
workflow_call:
inputs:
environment:
required: true
type: string
image-tag:
required: true
type: string
secrets:
KUBE_CONFIG:
required: true
jobs:
deploy:
runs-on: ubuntu-latest
environment: ${{ inputs.environment }}
steps:
- uses: actions/checkout@v4
- name: Set up kubectl
uses: azure/setup-kubectl@v3
- name: Configure kubeconfig
run: |
mkdir -p ~/.kube
echo "${{ secrets.KUBE_CONFIG }}" | base64 -d > ~/.kube/config
- name: Deploy
run: |
kubectl set image deployment/app \
app=${{ inputs.image-tag }} \
-n ${{ inputs.environment }}
kubectl rollout status deployment/app \
-n ${{ inputs.environment }} \
--timeout=300s
- name: Verify deployment
run: |
kubectl get pods -n ${{ inputs.environment }} -l app=app
kubectl logs -n ${{ inputs.environment }} -l app=app --tail=50undefinedname: Reusable Deploy Workflow
on:
workflow_call:
inputs:
environment:
required: true
type: string
image-tag:
required: true
type: string
secrets:
KUBE_CONFIG:
required: true
jobs:
deploy:
runs-on: ubuntu-latest
environment: ${{ inputs.environment }}
steps:
- uses: actions/checkout@v4
- name: Set up kubectl
uses: azure/setup-kubectl@v3
- name: Configure kubeconfig
run: |
mkdir -p ~/.kube
echo "${{ secrets.KUBE_CONFIG }}" | base64 -d > ~/.kube/config
- name: Deploy
run: |
kubectl set image deployment/app \
app=${{ inputs.image-tag }} \
-n ${{ inputs.environment }}
kubectl rollout status deployment/app \
-n ${{ inputs.environment }} \
--timeout=300s
- name: Verify deployment
run: |
kubectl get pods -n ${{ inputs.environment }} -l app=app
kubectl logs -n ${{ inputs.environment }} -l app=app --tail=50undefinedExample 4: Security Scanning Pipeline
示例4：安全扫描流水线
yaml
name: Security Scanning
on:
push:
branches: [main, develop]
pull_request:
branches: [main]
schedule:
- cron: "0 0 * * 0" # Weekly scan
jobs:
sast:
name: Static Analysis (SAST)
runs-on: ubuntu-latest
permissions:
security-events: write
contents: read
steps:
- uses: actions/checkout@v4
- name: Initialize CodeQL
uses: github/codeql-action/init@v3
with:
languages: javascript, python
- name: Autobuild
uses: github/codeql-action/autobuild@v3
- name: Perform CodeQL Analysis
uses: github/codeql-action/analyze@v3
- name: SonarCloud Scan
uses: SonarSource/sonarcloud-github-action@master
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}
with:
args: >
-Dsonar.organization=myorg
-Dsonar.projectKey=myproject
-Dsonar.qualitygate.wait=true
dependency-scan:
name: Dependency Vulnerability Scan
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Run Trivy vulnerability scanner
uses: aquasecurity/trivy-action@master
with:
scan-type: "fs"
scan-ref: "."
format: "sarif"
output: "trivy-results.sarif"
severity: "CRITICAL,HIGH"
- name: Upload Trivy results to GitHub Security
uses: github/codeql-action/upload-sarif@v3
with:
sarif_file: "trivy-results.sarif"
- name: Snyk Security Scan
uses: snyk/actions/node@master
env:
SNYK_TOKEN: ${{ secrets.SNYK_TOKEN }}
with:
args: --severity-threshold=high
secrets-scan:
name: Secrets Detection
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0 # Full history for secret detection
- name: TruffleHog Scan
uses: trufflesecurity/trufflehog@main
with:
path: ./
base: ${{ github.event.repository.default_branch }}
head: HEAD
- name: GitGuardian Scan
uses: GitGuardian/ggshield-action@v1
env:
GITHUB_PUSH_BEFORE_SHA: ${{ github.event.before }}
GITHUB_PUSH_BASE_SHA: ${{ github.event.base }}
GITHUB_DEFAULT_BRANCH: ${{ github.event.repository.default_branch }}
GITGUARDIAN_API_KEY: ${{ secrets.GITGUARDIAN_API_KEY }}
container-scan:
name: Container Image Scan
runs-on: ubuntu-latest
needs: [sast, dependency-scan]
steps:
- uses: actions/checkout@v4
- name: Build image
run: docker build -t myapp:${{ github.sha }} .
- name: Scan image with Trivy
uses: aquasecurity/trivy-action@master
with:
image-ref: "myapp:${{ github.sha }}"
format: "sarif"
output: "trivy-image-results.sarif"
- name: Scan image with Grype
uses: anchore/scan-action@v3
with:
image: "myapp:${{ github.sha }}"
fail-build: true
severity-cutoff: highyaml
name: Security Scanning
on:
push:
branches: [main, develop]
pull_request:
branches: [main]
schedule:
- cron: "0 0 * * 0" # Weekly scan
jobs:
sast:
name: Static Analysis (SAST)
runs-on: ubuntu-latest
permissions:
security-events: write
contents: read
steps:
- uses: actions/checkout@v4
- name: Initialize CodeQL
uses: github/codeql-action/init@v3
with:
languages: javascript, python
- name: Autobuild
uses: github/codeql-action/autobuild@v3
- name: Perform CodeQL Analysis
uses: github/codeql-action/analyze@v3
- name: SonarCloud Scan
uses: SonarSource/sonarcloud-github-action@master
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}
with:
args: >
-Dsonar.organization=myorg
-Dsonar.projectKey=myproject
-Dsonar.qualitygate.wait=true
dependency-scan:
name: Dependency Vulnerability Scan
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Run Trivy vulnerability scanner
uses: aquasecurity/trivy-action@master
with:
scan-type: "fs"
scan-ref: "."
format: "sarif"
output: "trivy-results.sarif"
severity: "CRITICAL,HIGH"
- name: Upload Trivy results to GitHub Security
uses: github/codeql-action/upload-sarif@v3
with:
sarif_file: "trivy-results.sarif"
- name: Snyk Security Scan
uses: snyk/actions/node@master
env:
SNYK_TOKEN: ${{ secrets.SNYK_TOKEN }}
with:
args: --severity-threshold=high
secrets-scan:
name: Secrets Detection
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0 # Full history for secret detection
- name: TruffleHog Scan
uses: trufflesecurity/trufflehog@main
with:
path: ./
base: ${{ github.event.repository.default_branch }}
head: HEAD
- name: GitGuardian Scan
uses: GitGuardian/ggshield-action@v1
env:
GITHUB_PUSH_BEFORE_SHA: ${{ github.event.before }}
GITHUB_PUSH_BASE_SHA: ${{ github.event.base }}
GITHUB_DEFAULT_BRANCH: ${{ github.event.repository.default_branch }}
GITGUARDIAN_API_KEY: ${{ secrets.GITGUARDIAN_API_KEY }}
container-scan:
name: Container Image Scan
runs-on: ubuntu-latest
needs: [sast, dependency-scan]
steps:
- uses: actions/checkout@v4
- name: Build image
run: docker build -t myapp:${{ github.sha }} .
- name: Scan image with Trivy
uses: aquasecurity/trivy-action@master
with:
image-ref: "myapp:${{ github.sha }}"
format: "sarif"
output: "trivy-image-results.sarif"
- name: Scan image with Grype
uses: anchore/scan-action@v3
with:
image: "myapp:${{ github.sha }}"
fail-build: true
severity-cutoff: highExample 5: Test Integration with Parallelization
示例5：带并行化的测试集成
yaml
name: Test Suite
on: [push, pull_request]
jobs:
unit-tests:
name: Unit Tests
runs-on: ubuntu-latest
strategy:
matrix:
node-version: [18, 20, 22]
os: [ubuntu-latest, macos-latest, windows-latest]
steps:
- uses: actions/checkout@v4
- name: Setup Node.js ${{ matrix.node-version }}
uses: actions/setup-node@v4
with:
node-version: ${{ matrix.node-version }}
cache: "npm"
- name: Install dependencies
run: npm ci
- name: Run unit tests
run: npm run test:unit -- --coverage --maxWorkers=4
- name: Upload coverage
uses: codecov/codecov-action@v3
with:
files: ./coverage/coverage-final.json
flags: unit-${{ matrix.os }}-node${{ matrix.node-version }}
integration-tests:
name: Integration Tests
runs-on: ubuntu-latest
strategy:
matrix:
shard: [1, 2, 3, 4]
services:
postgres:
image: postgres:16
env:
POSTGRES_PASSWORD: postgres
POSTGRES_DB: test
options: >-
--health-cmd pg_isready
--health-interval 10s
--health-timeout 5s
--health-retries 5
ports:
- 5432:5432
redis:
image: redis:7
options: >-
--health-cmd "redis-cli ping"
--health-interval 10s
--health-timeout 5s
--health-retries 5
ports:
- 6379:6379
steps:
- uses: actions/checkout@v4
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: 20
cache: "npm"
- name: Install dependencies
run: npm ci
- name: Run integration tests (shard ${{ matrix.shard }}/4)
run: npm run test:integration -- --shard=${{ matrix.shard }}/4
env:
DATABASE_URL: postgresql://postgres:postgres@localhost:5432/test
REDIS_URL: redis://localhost:6379
- name: Upload test results
if: always()
uses: actions/upload-artifact@v4
with:
name: integration-test-results-${{ matrix.shard }}
path: test-results/
e2e-tests:
name: E2E Tests
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: 20
cache: "npm"
- name: Install dependencies
run: npm ci
- name: Install Playwright
run: npx playwright install --with-deps
- name: Build application
run: npm run build
- name: Run E2E tests
run: npm run test:e2e
- name: Upload Playwright report
if: always()
uses: actions/upload-artifact@v4
with:
name: playwright-report
path: playwright-report/
test-report:
name: Generate Test Report
runs-on: ubuntu-latest
needs: [unit-tests, integration-tests, e2e-tests]
if: always()
steps:
- uses: actions/checkout@v4
- name: Download all test results
uses: actions/download-artifact@v4
with:
path: test-results/
- name: Generate combined report
run: |
npm install -g junit-viewer
junit-viewer --results=test-results/ --save=test-report.html
- name: Upload combined report
uses: actions/upload-artifact@v4
with:
name: combined-test-report
path: test-report.htmlyaml
name: Test Suite
on: [push, pull_request]
jobs:
unit-tests:
name: Unit Tests
runs-on: ubuntu-latest
strategy:
matrix:
node-version: [18, 20, 22]
os: [ubuntu-latest, macos-latest, windows-latest]
steps:
- uses: actions/checkout@v4
- name: Setup Node.js ${{ matrix.node-version }}
uses: actions/setup-node@v4
with:
node-version: ${{ matrix.node-version }}
cache: "npm"
- name: Install dependencies
run: npm ci
- name: Run unit tests
run: npm run test:unit -- --coverage --maxWorkers=4
- name: Upload coverage
uses: codecov/codecov-action@v3
with:
files: ./coverage/coverage-final.json
flags: unit-${{ matrix.os }}-node${{ matrix.node-version }}
integration-tests:
name: Integration Tests
runs-on: ubuntu-latest
strategy:
matrix:
shard: [1, 2, 3, 4]
services:
postgres:
image: postgres:16
env:
POSTGRES_PASSWORD: postgres
POSTGRES_DB: test
options: >-
--health-cmd pg_isready
--health-interval 10s
--health-timeout 5s
--health-retries 5
ports:
- 5432:5432
redis:
image: redis:7
options: >-
--health-cmd "redis-cli ping"
--health-interval 10s
--health-timeout 5s
--health-retries 5
ports:
- 6379:6379
steps:
- uses: actions/checkout@v4
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: 20
cache: "npm"
- name: Install dependencies
run: npm ci
- name: Run integration tests (shard ${{ matrix.shard }}/4)
run: npm run test:integration -- --shard=${{ matrix.shard }}/4
env:
DATABASE_URL: postgresql://postgres:postgres@localhost:5432/test
REDIS_URL: redis://localhost:6379
- name: Upload test results
if: always()
uses: actions/upload-artifact@v4
with:
name: integration-test-results-${{ matrix.shard }}
path: test-results/
e2e-tests:
name: E2E Tests
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: 20
cache: "npm"
- name: Install dependencies
run: npm ci
- name: Install Playwright
run: npx playwright install --with-deps
- name: Build application
run: npm run build
- name: Run E2E tests
run: npm run test:e2e
- name: Upload Playwright report
if: always()
uses: actions/upload-artifact@v4
with:
name: playwright-report
path: playwright-report/
test-report:
name: Generate Test Report
runs-on: ubuntu-latest
needs: [unit-tests, integration-tests, e2e-tests]
if: always()
steps:
- uses: actions/checkout@v4
- name: Download all test results
uses: actions/download-artifact@v4
with:
path: test-results/
- name: Generate combined report
run: |
npm install -g junit-viewer
junit-viewer --results=test-results/ --save=test-report.html
- name: Upload combined report
uses: actions/upload-artifact@v4
with:
name: combined-test-report
path: test-report.htmlExample 6: ML Pipeline (Model Training & Deployment)
示例6：机器学习流水线（模型训练与部署）
yaml
name: ML Pipeline
on:
push:
branches: [main]
paths:
- "models/**"
- "training/**"
- "data/**"
workflow_dispatch:
inputs:
model-version:
description: "Model version to train"
required: true
type: string
jobs:
data-validation:
name: Validate Training Data
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Setup Python
uses: actions/setup-python@v5
with:
python-version: "3.11"
cache: "pip"
- name: Install dependencies
run: |
pip install pandas great-expectations dvc
- name: Pull data with DVC
run: |
dvc remote modify origin --local auth basic
dvc remote modify origin --local user ${{ secrets.DVC_USER }}
dvc remote modify origin --local password ${{ secrets.DVC_PASSWORD }}
dvc pull
- name: Validate data schema
run: python scripts/validate_data.py
- name: Run Great Expectations
run: great_expectations checkpoint run training_data_checkpoint
train-model:
name: Train ML Model
runs-on: ubuntu-latest
needs: data-validation
steps:
- uses: actions/checkout@v4
- name: Setup Python
uses: actions/setup-python@v5
with:
python-version: "3.11"
cache: "pip"
- name: Install dependencies
run: |
pip install -r requirements.txt
pip install mlflow wandb
- name: Configure MLflow
run: |
echo "MLFLOW_TRACKING_URI=${{ secrets.MLFLOW_TRACKING_URI }}" >> $GITHUB_ENV
echo "MLFLOW_TRACKING_USERNAME=${{ secrets.MLFLOW_USERNAME }}" >> $GITHUB_ENV
echo "MLFLOW_TRACKING_PASSWORD=${{ secrets.MLFLOW_PASSWORD }}" >> $GITHUB_ENV
- name: Train model
run: |
python training/train.py \
--experiment-name "prod-training" \
--model-version ${{ inputs.model-version || github.sha }} \
--config training/config.yaml
env:
WANDB_API_KEY: ${{ secrets.WANDB_API_KEY }}
- name: Upload model artifact
uses: actions/upload-artifact@v4
with:
name: trained-model
path: models/output/
evaluate-model:
name: Evaluate Model Performance
runs-on: ubuntu-latest
needs: train-model
steps:
- uses: actions/checkout@v4
- name: Setup Python
uses: actions/setup-python@v5
with:
python-version: "3.11"
cache: "pip"
- name: Install dependencies
run: pip install -r requirements.txt
- name: Download model
uses: actions/download-artifact@v4
with:
name: trained-model
path: models/output/
- name: Run model evaluation
run: python evaluation/evaluate.py --model-path models/output/
- name: Check performance thresholds
run: |
python evaluation/check_metrics.py \
--min-accuracy 0.85 \
--min-f1 0.80
- name: Generate model card
run: python scripts/generate_model_card.py
deploy-model:
name: Deploy Model to Production
runs-on: ubuntu-latest
needs: evaluate-model
if: github.ref == 'refs/heads/main'
environment: production
steps:
- uses: actions/checkout@v4
- name: Download model
uses: actions/download-artifact@v4
with:
name: trained-model
path: models/output/
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@v4
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: us-east-1
- name: Upload model to S3
run: |
aws s3 cp models/output/model.pkl \
s3://my-ml-models/prod/${{ github.sha }}/model.pkl
- name: Deploy to SageMaker
run: |
python deployment/deploy_sagemaker.py \
--model-uri s3://my-ml-models/prod/${{ github.sha }}/model.pkl \
--endpoint-name prod-ml-endpoint \
--instance-type ml.m5.large
- name: Run smoke tests
run: python deployment/smoke_test.py --endpoint prod-ml-endpoint
- name: Update model registry
run: |
python scripts/register_model.py \
--version ${{ github.sha }} \
--stage production \
--metadata models/output/metadata.jsonyaml
name: ML Pipeline
on:
push:
branches: [main]
paths:
- "models/**"
- "training/**"
- "data/**"
workflow_dispatch:
inputs:
model-version:
description: "Model version to train"
required: true
type: string
jobs:
data-validation:
name: Validate Training Data
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Setup Python
uses: actions/setup-python@v5
with:
python-version: "3.11"
cache: "pip"
- name: Install dependencies
run: |
pip install pandas great-expectations dvc
- name: Pull data with DVC
run: |
dvc remote modify origin --local auth basic
dvc remote modify origin --local user ${{ secrets.DVC_USER }}
dvc remote modify origin --local password ${{ secrets.DVC_PASSWORD }}
dvc pull
- name: Validate data schema
run: python scripts/validate_data.py
- name: Run Great Expectations
run: great_expectations checkpoint run training_data_checkpoint
train-model:
name: Train ML Model
runs-on: ubuntu-latest
needs: data-validation
steps:
- uses: actions/checkout@v4
- name: Setup Python
uses: actions/setup-python@v5
with:
python-version: "3.11"
cache: "pip"
- name: Install dependencies
run: |
pip install -r requirements.txt
pip install mlflow wandb
- name: Configure MLflow
run: |
echo "MLFLOW_TRACKING_URI=${{ secrets.MLFLOW_TRACKING_URI }}" >> $GITHUB_ENV
echo "MLFLOW_TRACKING_USERNAME=${{ secrets.MLFLOW_USERNAME }}" >> $GITHUB_ENV
echo "MLFLOW_TRACKING_PASSWORD=${{ secrets.MLFLOW_PASSWORD }}" >> $GITHUB_ENV
- name: Train model
run: |
python training/train.py \
--experiment-name "prod-training" \
--model-version ${{ inputs.model-version || github.sha }} \
--config training/config.yaml
env:
WANDB_API_KEY: ${{ secrets.WANDB_API_KEY }}
- name: Upload model artifact
uses: actions/upload-artifact@v4
with:
name: trained-model
path: models/output/
evaluate-model:
name: Evaluate Model Performance
runs-on: ubuntu-latest
needs: train-model
steps:
- uses: actions/checkout@v4
- name: Setup Python
uses: actions/setup-python@v5
with:
python-version: "3.11"
cache: "pip"
- name: Install dependencies
run: pip install -r requirements.txt
- name: Download model
uses: actions/download-artifact@v4
with:
name: trained-model
path: models/output/
- name: Run model evaluation
run: python evaluation/evaluate.py --model-path models/output/
- name: Check performance thresholds
run: |
python evaluation/check_metrics.py \
--min-accuracy 0.85 \
--min-f1 0.80
- name: Generate model card
run: python scripts/generate_model_card.py
deploy-model:
name: Deploy Model to Production
runs-on: ubuntu-latest
needs: evaluate-model
if: github.ref == 'refs/heads/main'
environment: production
steps:
- uses: actions/checkout@v4
- name: Download model
uses: actions/download-artifact@v4
with:
name: trained-model
path: models/output/
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@v4
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: us-east-1
- name: Upload model to S3
run: |
aws s3 cp models/output/model.pkl \
s3://my-ml-models/prod/${{ github.sha }}/model.pkl
- name: Deploy to SageMaker
run: |
python deployment/deploy_sagemaker.py \
--model-uri s3://my-ml-models/prod/${{ github.sha }}/model.pkl \
--endpoint-name prod-ml-endpoint \
--instance-type ml.m5.large
- name: Run smoke tests
run: python deployment/smoke_test.py --endpoint prod-ml-endpoint
- name: Update model registry
run: |
python scripts/register_model.py \
--version ${{ github.sha }} \
--stage production \
--metadata models/output/metadata.jsonExample 7: Monorepo CI with Path Filtering
示例7：带路径过滤的单体仓库CI
yaml
name: Monorepo CI
on:
push:
branches: [main, develop]
pull_request:
branches: [main]
jobs:
detect-changes:
name: Detect Changed Services
runs-on: ubuntu-latest
outputs:
api: ${{ steps.filter.outputs.api }}
web: ${{ steps.filter.outputs.web }}
worker: ${{ steps.filter.outputs.worker }}
shared: ${{ steps.filter.outputs.shared }}
steps:
- uses: actions/checkout@v4
- uses: dorny/paths-filter@v3
id: filter
with:
filters: |
api:
- 'services/api/**'
- 'packages/shared/**'
web:
- 'services/web/**'
- 'packages/shared/**'
worker:
- 'services/worker/**'
- 'packages/shared/**'
shared:
- 'packages/shared/**'
test-api:
name: Test API Service
needs: detect-changes
if: needs.detect-changes.outputs.api == 'true'
runs-on: ubuntu-latest
defaults:
run:
working-directory: services/api
steps:
- uses: actions/checkout@v4
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: 20
cache: "npm"
cache-dependency-path: services/api/package-lock.json
- name: Install dependencies
run: npm ci
- name: Run tests
run: npm test
test-web:
name: Test Web Service
needs: detect-changes
if: needs.detect-changes.outputs.web == 'true'
runs-on: ubuntu-latest
defaults:
run:
working-directory: services/web
steps:
- uses: actions/checkout@v4
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: 20
cache: "npm"
cache-dependency-path: services/web/package-lock.json
- name: Install dependencies
run: npm ci
- name: Run tests
run: npm test
- name: Build
run: npm run build
test-worker:
name: Test Worker Service
needs: detect-changes
if: needs.detect-changes.outputs.worker == 'true'
runs-on: ubuntu-latest
defaults:
run:
working-directory: services/worker
steps:
- uses: actions/checkout@v4
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: 20
cache: "npm"
cache-dependency-path: services/worker/package-lock.json
- name: Install dependencies
run: npm ci
- name: Run tests
run: npm test
build-and-deploy:
name: Build and Deploy Changed Services
needs: [detect-changes, test-api, test-web, test-worker]
if: |
always() &&
(needs.test-api.result == 'success' || needs.test-api.result == 'skipped') &&
(needs.test-web.result == 'success' || needs.test-web.result == 'skipped') &&
(needs.test-worker.result == 'success' || needs.test-worker.result == 'skipped')
runs-on: ubuntu-latest
strategy:
matrix:
service:
- name: api
changed: ${{ needs.detect-changes.outputs.api == 'true' }}
- name: web
changed: ${{ needs.detect-changes.outputs.web == 'true' }}
- name: worker
changed: ${{ needs.detect-changes.outputs.worker == 'true' }}
steps:
- uses: actions/checkout@v4
if: matrix.service.changed == 'true'
- name: Build and push ${{ matrix.service.name }}
if: matrix.service.changed == 'true'
run: |
docker build -t myapp-${{ matrix.service.name }}:${{ github.sha }} \
services/${{ matrix.service.name }}
docker push myapp-${{ matrix.service.name }}:${{ github.sha }}yaml
name: Monorepo CI
on:
push:
branches: [main, develop]
pull_request:
branches: [main]
jobs:
detect-changes:
name: Detect Changed Services
runs-on: ubuntu-latest
outputs:
api: ${{ steps.filter.outputs.api }}
web: ${{ steps.filter.outputs.web }}
worker: ${{ steps.filter.outputs.worker }}
shared: ${{ steps.filter.outputs.shared }}
steps:
- uses: actions/checkout@v4
- uses: dorny/paths-filter@v3
id: filter
with:
filters: |
api:
- 'services/api/**'
- 'packages/shared/**'
web:
- 'services/web/**'
- 'packages/shared/**'
worker:
- 'services/worker/**'
- 'packages/shared/**'
shared:
- 'packages/shared/**'
test-api:
name: Test API Service
needs: detect-changes
if: needs.detect-changes.outputs.api == 'true'
runs-on: ubuntu-latest
defaults:
run:
working-directory: services/api
steps:
- uses: actions/checkout@v4
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: 20
cache: "npm"
cache-dependency-path: services/api/package-lock.json
- name: Install dependencies
run: npm ci
- name: Run tests
run: npm test
test-web:
name: Test Web Service
needs: detect-changes
if: needs.detect-changes.outputs.web == 'true'
runs-on: ubuntu-latest
defaults:
run:
working-directory: services/web
steps:
- uses: actions/checkout@v4
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: 20
cache: "npm"
cache-dependency-path: services/web/package-lock.json
- name: Install dependencies
run: npm ci
- name: Run tests
run: npm test
- name: Build
run: npm run build
test-worker:
name: Test Worker Service
needs: detect-changes
if: needs.detect-changes.outputs.worker == 'true'
runs-on: ubuntu-latest
defaults:
run:
working-directory: services/worker
steps:
- uses: actions/checkout@v4
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: 20
cache: "npm"
cache-dependency-path: services/worker/package-lock.json
- name: Install dependencies
run: npm ci
- name: Run tests
run: npm test
build-and-deploy:
name: Build and Deploy Changed Services
needs: [detect-changes, test-api, test-web, test-worker]
if: |
always() &&
(needs.test-api.result == 'success' || needs.test-api.result == 'skipped') &&
(needs.test-web.result == 'success' || needs.test-web.result == 'skipped') &&
(needs.test-worker.result == 'success' || needs.test-worker.result == 'skipped')
runs-on: ubuntu-latest
strategy:
matrix:
service:
- name: api
changed: ${{ needs.detect-changes.outputs.api == 'true' }}
- name: web
changed: ${{ needs.detect-changes.outputs.web == 'true' }}
- name: worker
changed: ${{ needs.detect-changes.outputs.worker == 'true' }}
steps:
- uses: actions/checkout@v4
if: matrix.service.changed == 'true'
- name: Build and push ${{ matrix.service.name }}
if: matrix.service.changed == 'true'
run: |
docker build -t myapp-${{ matrix.service.name }}:${{ github.sha }} \
services/${{ matrix.service.name }}
docker push myapp-${{ matrix.service.name }}:${{ github.sha }}Example 8: Performance Optimization Pipeline
示例8：性能优化流水线
yaml
name: Optimized CI Pipeline
on: [push, pull_request]
jobs:
# Fast feedback jobs run first
quick-checks:
name: Quick Checks (< 2min)
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: 20
cache: "npm"
- name: Cache node_modules
uses: actions/cache@v4
with:
path: node_modules
key: ${{ runner.os }}-node-${{ hashFiles('**/package-lock.json') }}
restore-keys: |
${{ runner.os }}-node-
- name: Install dependencies
run: npm ci --prefer-offline --no-audit
- name: Parallel lint and type check
run: |
npm run lint &
npm run type-check &
wait
unit-tests-fast:
name: Unit Tests (Changed Files Only)
runs-on: ubuntu-latest
if: github.event_name == 'pull_request'
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0 # Need full history for changed files
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: 20
cache: "npm"
- name: Install dependencies
run: npm ci --prefer-offline
- name: Get changed files
id: changed-files
run: |
echo "files=$(git diff --name-only origin/main...HEAD | \
grep -E '\.(ts|tsx|js|jsx)$' | \
xargs -I {} echo '--findRelatedTests {}' | \
tr '\n' ' ')" >> $GITHUB_OUTPUT
- name: Run tests for changed files only
if: steps.changed-files.outputs.files != ''
run: npm test -- ${{ steps.changed-files.outputs.files }}
build-with-cache:
name: Build with Aggressive Caching
runs-on: ubuntu-latest
needs: quick-checks
steps:
- uses: actions/checkout@v4
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: 20
cache: "npm"
- name: Cache build output
uses: actions/cache@v4
with:
path: |
.next/cache
dist/
build/
key: ${{ runner.os }}-build-${{ hashFiles('**/*.ts', '**/*.tsx', '**/*.js') }}
restore-keys: |
${{ runner.os }}-build-
- name: Install dependencies
run: npm ci --prefer-offline
- name: Build
run: npm run build
- name: Upload build artifacts
uses: actions/upload-artifact@v4
with:
name: build-output
path: dist/
retention-days: 7
docker-build-optimized:
name: Docker Build with Layer Caching
runs-on: ubuntu-latest
needs: quick-checks
steps:
- uses: actions/checkout@v4
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Build with cache
uses: docker/build-push-action@v5
with:
context: .
push: false
tags: myapp:${{ github.sha }}
cache-from: type=gha
cache-to: type=gha,mode=max
build-args: |
BUILDKIT_INLINE_CACHE=1yaml
name: Optimized CI Pipeline
on: [push, pull_request]
jobs:
# Fast feedback jobs run first
quick-checks:
name: Quick Checks (< 2min)
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: 20
cache: "npm"
- name: Cache node_modules
uses: actions/cache@v4
with:
path: node_modules
key: ${{ runner.os }}-node-${{ hashFiles('**/package-lock.json') }}
restore-keys: |
${{ runner.os }}-node-
- name: Install dependencies
run: npm ci --prefer-offline --no-audit
- name: Parallel lint and type check
run: |
npm run lint &
npm run type-check &
wait
unit-tests-fast:
name: Unit Tests (Changed Files Only)
runs-on: ubuntu-latest
if: github.event_name == 'pull_request'
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0 # Need full history for changed files
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: 20
cache: "npm"
- name: Install dependencies
run: npm ci --prefer-offline
- name: Get changed files
id: changed-files
run: |
echo "files=$(git diff --name-only origin/main...HEAD | \
grep -E '\.(ts|tsx|js|jsx)$' | \
xargs -I {} echo '--findRelatedTests {}' | \
tr '\n' ' ')" >> $GITHUB_OUTPUT
- name: Run tests for changed files only
if: steps.changed-files.outputs.files != ''
run: npm test -- ${{ steps.changed-files.outputs.files }}
build-with-cache:
name: Build with Aggressive Caching
runs-on: ubuntu-latest
needs: quick-checks
steps:
- uses: actions/checkout@v4
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: 20
cache: "npm"
- name: Cache build output
uses: actions/cache@v4
with:
path: |
.next/cache
dist/
build/
key: ${{ runner.os }}-build-${{ hashFiles('**/*.ts', '**/*.tsx', '**/*.js') }}
restore-keys: |
${{ runner.os }}-build-
- name: Install dependencies
run: npm ci --prefer-offline
- name: Build
run: npm run build
- name: Upload build artifacts
uses: actions/upload-artifact@v4
with:
name: build-output
path: dist/
retention-days: 7
docker-build-optimized:
name: Docker Build with Layer Caching
runs-on: ubuntu-latest
needs: quick-checks
steps:
- uses: actions/checkout@v4
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Build with cache
uses: docker/build-push-action@v5
with:
context: .
push: false
tags: myapp:${{ github.sha }}
cache-from: type=gha
cache-to: type=gha,mode=max
build-args: |
BUILDKIT_INLINE_CACHE=1Pipeline Optimization Patterns
流水线优化模式
Caching Strategy
缓存策略
- Dependency Caching: Cache ,
node_modules,vendor/, etc..m2/ - Build Output Caching: Cache compiled artifacts between runs
- Docker Layer Caching: Use BuildKit cache mounts and GitHub Actions cache
- Incremental Builds: Only rebuild changed modules (Nx, Turborepo)
- 依赖缓存：缓存、
node_modules、vendor/等.m2/ - 构建输出缓存：在多次运行之间缓存编译后的制品
- Docker分层缓存：使用BuildKit缓存挂载与GitHub Actions缓存
- 增量构建：仅重新构建变更的模块（Nx、Turborepo）
Parallelization Strategies
并行化策略
- Job-Level Parallelization: Run independent jobs concurrently
- Test Sharding: Split test suite across multiple runners
- Matrix Builds: Test multiple versions/platforms simultaneously
- Monorepo Path Filtering: Only test changed services
- 作业级并行化：并发执行独立作业
- 测试分片：将测试套件拆分到多个运行器上执行
- 矩阵构建：同时测试多个版本/平台
- 单体仓库路径过滤：仅测试变更的服务
Conditional Execution
条件执行
- Path Filters: Skip jobs when irrelevant files change
- Changed Files Detection: Test only affected code
- Branch-Specific Jobs: Different pipelines for different branches
- Manual Triggers: Allow on-demand pipeline execution
- 路径过滤：无关文件变更时跳过作业
- 变更文件检测：仅测试受影响的代码
- 分支专属作业：为不同分支配置不同的流水线
- 手动触发：允许按需执行流水线
ML-Specific Patterns
机器学习专属模式
Model Training Pipeline
模型训练流水线
- Data Validation: Validate schema and quality before training
- Experiment Tracking: Log metrics to MLflow/W&B
- Model Versioning: Tag models with git SHA or semantic version
- Performance Gates: Enforce minimum accuracy/F1 thresholds
- 数据验证：训练前验证数据 schema 与质量
- 实验追踪：将指标记录到MLflow/W&B
- 模型版本控制：使用Git SHA或语义版本为模型打标签
- 性能门禁：强制执行最低准确率/F1阈值
Model Deployment
模型部署
- A/B Testing: Deploy new model alongside existing
- Shadow Mode: Run new model without affecting production
- Canary Rollout: Gradually increase traffic to new model
- Automated Rollback: Revert on performance degradation
- A/B测试：将新模型与现有模型并行部署
- 影子模式：运行新模型但不影响生产流量
- 金丝雀发布：逐步增加新模型的流量占比
- 自动回滚：性能下降时自动回滚
Troubleshooting Guide
排查指南
Common Issues
常见问题
- Flaky Tests: Implement retry logic, increase timeouts, fix race conditions
- Slow Pipelines: Profile execution times, add caching, parallelize
- Secrets Exposure: Use secret scanning, audit logs, rotate credentials
- Resource Exhaustion: Set resource limits, use cleanup actions
- Network Timeouts: Add retries, use artifact caching, increase timeouts
- 不稳定测试：实现重试逻辑、增加超时时间、修复竞态条件
- 缓慢流水线：分析执行时间、添加缓存、并行化作业
- 密钥泄露：使用密钥扫描、审计日志、轮换凭证
- 资源耗尽：设置资源限制、使用清理操作
- 网络超时：添加重试、使用制品缓存、增加超时时间
Debugging Commands
调试命令
bash
undefinedbash
undefinedGitHub Actions local testing
GitHub Actions 本地测试
act -j test --secret-file .env.secrets
act -j test --secret-file .env.secrets
GitLab CI local testing
GitLab CI 本地测试
gitlab-runner exec docker test
gitlab-runner exec docker test
Jenkins pipeline validation
Jenkins 流水线验证
java -jar jenkins-cli.jar declarative-linter < Jenkinsfile
java -jar jenkins-cli.jar declarative-linter < Jenkinsfile
Docker build debugging
Docker 构建调试
DOCKER_BUILDKIT=1 docker build --progress=plain .
DOCKER_BUILDKIT=1 docker build --progress=plain .
Test pipeline with dry-run
测试流水线 dry-run
kubectl apply --dry-run=client -f k8s/
kubectl apply --dry-run=client -f k8s/
Validate workflow syntax
验证工作流语法
actionlint .github/workflows/*.yml
undefinedactionlint .github/workflows/*.yml
undefined