monorepo-ci-optimizer

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

Monorepo CI Optimizer

Monorepo CI 优化器

Run only affected builds and tests in monorepos for faster CI.
在单体仓库中仅运行受影响代码的构建和测试,以加速CI流程。

Affected Detection Strategy

受影响代码检测策略

Using Turborepo

使用Turborepo

yaml
undefined
yaml
undefined

.github/workflows/ci.yml

.github/workflows/ci.yml

name: CI
on: pull_request: push: branches: [main]
jobs: build: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 with: fetch-depth: 0 # Need full history for affected detection
  - uses: actions/setup-node@v4
    with:
      node-version: "20"
      cache: "pnpm"

  - run: pnpm install

  - name: Build affected
    run: |
      pnpm turbo run build --filter='...[origin/main]'

  - name: Test affected
    run: |
      pnpm turbo run test --filter='...[origin/main]'

  - name: Lint affected
    run: |
      pnpm turbo run lint --filter='...[origin/main]'
undefined
name: CI
on: pull_request: push: branches: [main]
jobs: build: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 with: fetch-depth: 0 # Need full history for affected detection
  - uses: actions/setup-node@v4
    with:
      node-version: "20"
      cache: "pnpm"

  - run: pnpm install

  - name: Build affected
    run: |
      pnpm turbo run build --filter='...[origin/main]'

  - name: Test affected
    run: |
      pnpm turbo run test --filter='...[origin/main]'

  - name: Lint affected
    run: |
      pnpm turbo run lint --filter='...[origin/main]'
undefined

Using Nx

使用Nx

yaml
build:
  runs-on: ubuntu-latest
  steps:
    - uses: actions/checkout@v4
      with:
        fetch-depth: 0

    - uses: nrwl/nx-set-shas@v4

    - uses: actions/setup-node@v4
      with:
        node-version: "20"
        cache: "npm"

    - run: npm ci

    - name: Build affected projects
      run: npx nx affected:build --base=$NX_BASE --head=$NX_HEAD

    - name: Test affected projects
      run: npx nx affected:test --base=$NX_BASE --head=$NX_HEAD --parallel=3

    - name: Lint affected projects
      run: npx nx affected:lint --base=$NX_BASE --head=$NX_HEAD
yaml
build:
  runs-on: ubuntu-latest
  steps:
    - uses: actions/checkout@v4
      with:
        fetch-depth: 0

    - uses: nrwl/nx-set-shas@v4

    - uses: actions/setup-node@v4
      with:
        node-version: "20"
        cache: "npm"

    - run: npm ci

    - name: Build affected projects
      run: npx nx affected:build --base=$NX_BASE --head=$NX_HEAD

    - name: Test affected projects
      run: npx nx affected:test --base=$NX_BASE --head=$NX_HEAD --parallel=3

    - name: Lint affected projects
      run: npx nx affected:lint --base=$NX_BASE --head=$NX_HEAD

Remote Caching (Turborepo)

远程缓存(Turborepo)

yaml
- name: Setup Turborepo cache
  uses: actions/cache@v3
  with:
    path: .turbo
    key: ${{ runner.os }}-turbo-${{ github.sha }}
    restore-keys: |
      ${{ runner.os }}-turbo-

- name: Build with remote cache
  run: pnpm turbo run build
  env:
    TURBO_TOKEN: ${{ secrets.TURBO_TOKEN }}
    TURBO_TEAM: ${{ secrets.TURBO_TEAM }}
yaml
- name: Setup Turborepo cache
  uses: actions/cache@v3
  with:
    path: .turbo
    key: ${{ runner.os }}-turbo-${{ github.sha }}
    restore-keys: |
      ${{ runner.os }}-turbo-

- name: Build with remote cache
  run: pnpm turbo run build
  env:
    TURBO_TOKEN: ${{ secrets.TURBO_TOKEN }}
    TURBO_TEAM: ${{ secrets.TURBO_TEAM }}

Nx Cloud Integration

Nx Cloud 集成

yaml
- name: Setup Nx Cloud
  run: |
    npx nx-cloud start-ci-run --distribute-on="3 linux-medium-js"
  env:
    NX_CLOUD_ACCESS_TOKEN: ${{ secrets.NX_CLOUD_ACCESS_TOKEN }}

- name: Run affected
  run: |
    npx nx affected --target=build --parallel=3
    npx nx affected --target=test --parallel=3
    npx nx affected --target=lint --parallel=3
yaml
- name: Setup Nx Cloud
  run: |
    npx nx-cloud start-ci-run --distribute-on="3 linux-medium-js"
  env:
    NX_CLOUD_ACCESS_TOKEN: ${{ secrets.NX_CLOUD_ACCESS_TOKEN }}

- name: Run affected
  run: |
    npx nx affected --target=build --parallel=3
    npx nx affected --target=test --parallel=3
    npx nx affected --target=lint --parallel=3

Manual Affected Detection

手动受影响代码检测

typescript
// scripts/get-affected.ts
import * as path from "path";
import { execSync } from "child_process";
import * as fs from "fs";

interface Package {
  name: string;
  path: string;
  dependencies: string[];
}

function getChangedFiles(base: string = "origin/main"): string[] {
  const output = execSync(`git diff --name-only ${base}...HEAD`, {
    encoding: "utf-8",
  });
  return output.split("\n").filter(Boolean);
}

function getPackages(): Package[] {
  const packagesDir = path.join(process.cwd(), "packages");
  return fs.readdirSync(packagesDir).map((name) => {
    const packageJson = JSON.parse(
      fs.readFileSync(path.join(packagesDir, name, "package.json"), "utf-8")
    );
    return {
      name: packageJson.name,
      path: `packages/${name}`,
      dependencies: [
        ...Object.keys(packageJson.dependencies || {}),
        ...Object.keys(packageJson.devDependencies || {}),
      ],
    };
  });
}

function getAffectedPackages(): string[] {
  const changedFiles = getChangedFiles();
  const packages = getPackages();
  const affected = new Set<string>();

  // Direct changes
  changedFiles.forEach((file) => {
    packages.forEach((pkg) => {
      if (file.startsWith(pkg.path)) {
        affected.add(pkg.name);
      }
    });
  });

  // Dependent packages
  let changed = true;
  while (changed) {
    changed = false;
    packages.forEach((pkg) => {
      if (!affected.has(pkg.name)) {
        const hasAffectedDep = pkg.dependencies.some((dep) =>
          affected.has(dep)
        );
        if (hasAffectedDep) {
          affected.add(pkg.name);
          changed = true;
        }
      }
    });
  }

  return Array.from(affected);
}

// Output for GitHub Actions
const affected = getAffectedPackages();
console.log(`::set-output name=packages::${affected.join(",")}`);
console.log(`Affected packages: ${affected.join(", ")}`);
typescript
// scripts/get-affected.ts
import * as path from "path";
import { execSync } from "child_process";
import * as fs from "fs";

interface Package {
  name: string;
  path: string;
  dependencies: string[];
}

function getChangedFiles(base: string = "origin/main"): string[] {
  const output = execSync(`git diff --name-only ${base}...HEAD`, {
    encoding: "utf-8",
  });
  return output.split("\n").filter(Boolean);
}

function getPackages(): Package[] {
  const packagesDir = path.join(process.cwd(), "packages");
  return fs.readdirSync(packagesDir).map((name) => {
    const packageJson = JSON.parse(
      fs.readFileSync(path.join(packagesDir, name, "package.json"), "utf-8")
    );
    return {
      name: packageJson.name,
      path: `packages/${name}`,
      dependencies: [
        ...Object.keys(packageJson.dependencies || {}),
        ...Object.keys(packageJson.devDependencies || {}),
      ],
    };
  });
}

function getAffectedPackages(): string[] {
  const changedFiles = getChangedFiles();
  const packages = getPackages();
  const affected = new Set<string>();

  // Direct changes
  changedFiles.forEach((file) => {
    packages.forEach((pkg) => {
      if (file.startsWith(pkg.path)) {
        affected.add(pkg.name);
      }
    });
  });

  // Dependent packages
  let changed = true;
  while (changed) {
    changed = false;
    packages.forEach((pkg) => {
      if (!affected.has(pkg.name)) {
        const hasAffectedDep = pkg.dependencies.some((dep) =>
          affected.has(dep)
        );
        if (hasAffectedDep) {
          affected.add(pkg.name);
          changed = true;
        }
      }
    });
  }

  return Array.from(affected);
}

// Output for GitHub Actions
const affected = getAffectedPackages();
console.log(`::set-output name=packages::${affected.join(",")}`);
console.log(`Affected packages: ${affected.join(", ")}`);

Matrix Strategy for Affected Packages

针对受影响包的矩阵策略

yaml
detect-affected:
  runs-on: ubuntu-latest
  outputs:
    packages: ${{ steps.affected.outputs.packages }}
  steps:
    - uses: actions/checkout@v4
      with:
        fetch-depth: 0

    - id: affected
      run: |
        PACKAGES=$(node scripts/get-affected.ts)
        echo "packages=$PACKAGES" >> $GITHUB_OUTPUT

test-affected:
  needs: detect-affected
  if: needs.detect-affected.outputs.packages != ''
  runs-on: ubuntu-latest
  strategy:
    matrix:
      package: ${{ fromJSON(needs.detect-affected.outputs.packages) }}
  steps:
    - uses: actions/checkout@v4

    - uses: actions/setup-node@v4
      with:
        node-version: "20"
        cache: "pnpm"

    - run: pnpm install

    - name: Test ${{ matrix.package }}
      run: pnpm --filter ${{ matrix.package }} test
yaml
detect-affected:
  runs-on: ubuntu-latest
  outputs:
    packages: ${{ steps.affected.outputs.packages }}
  steps:
    - uses: actions/checkout@v4
      with:
        fetch-depth: 0

    - id: affected
      run: |
        PACKAGES=$(node scripts/get-affected.ts)
        echo "packages=$PACKAGES" >> $GITHUB_OUTPUT

test-affected:
  needs: detect-affected
  if: needs.detect-affected.outputs.packages != ''
  runs-on: ubuntu-latest
  strategy:
    matrix:
      package: ${{ fromJSON(needs.detect-affected.outputs.packages) }}
  steps:
    - uses: actions/checkout@v4

    - uses: actions/setup-node@v4
      with:
        node-version: "20"
        cache: "pnpm"

    - run: pnpm install

    - name: Test ${{ matrix.package }}
      run: pnpm --filter ${{ matrix.package }} test

Workspace-aware Caching

工作区感知缓存

yaml
- name: Cache workspace builds
  uses: actions/cache@v3
  with:
    path: |
      packages/*/dist
      packages/*/node_modules/.cache
    key: ${{ runner.os }}-workspace-${{ hashFiles('**/pnpm-lock.yaml') }}-${{ hashFiles('packages/**/*.ts') }}
    restore-keys: |
      ${{ runner.os }}-workspace-${{ hashFiles('**/pnpm-lock.yaml') }}-
      ${{ runner.os }}-workspace-
yaml
- name: Cache workspace builds
  uses: actions/cache@v3
  with:
    path: |
      packages/*/dist
      packages/*/node_modules/.cache
    key: ${{ runner.os }}-workspace-${{ hashFiles('**/pnpm-lock.yaml') }}-${{ hashFiles('packages/**/*.ts') }}
    restore-keys: |
      ${{ runner.os }}-workspace-${{ hashFiles('**/pnpm-lock.yaml') }}-
      ${{ runner.os }}-workspace-

Parallel Execution

并行执行

yaml
- name: Build all packages in parallel
  run: pnpm turbo run build --parallel

- name: Test with controlled parallelism
  run: pnpm turbo run test --concurrency=3
yaml
- name: Build all packages in parallel
  run: pnpm turbo run build --parallel

- name: Test with controlled parallelism
  run: pnpm turbo run test --concurrency=3

Optimization Metrics

优化指标

markdown
undefined
markdown
undefined

Before Optimization

优化前

  • Full build: 25 minutes
  • All tests: 15 minutes
  • Total: 40 minutes
  • Runs on every PR
  • 完整构建:25分钟
  • 全部测试:15分钟
  • 总计:40分钟
  • 每个PR都会运行

After Affected Detection

启用受影响代码检测后

  • Affected build: 5 minutes (80% reduction)
  • Affected tests: 3 minutes (80% reduction)
  • Total: 8 minutes (80% reduction)
  • Only runs necessary checks
  • 受影响代码构建:5分钟(减少80%)
  • 受影响代码测试:3分钟(减少80%)
  • 总计:8分钟(减少80%)
  • 仅运行必要的检查

With Remote Caching

启用远程缓存后

  • Cache hit build: 30 seconds (98% reduction)
  • Cache hit tests: 1 minute (93% reduction)
  • Total: 1.5 minutes (96% reduction)
undefined
  • 缓存命中构建:30秒(减少98%)
  • 缓存命中测试:1分钟(减少93%)
  • 总计:1.5分钟(减少96%)
undefined

Best Practices

最佳实践

  1. Fetch full history:
    fetch-depth: 0
    for accurate diffs
  2. Topological order: Build dependencies first
  3. Remote caching: Share cache across CI runs
  4. Parallel execution: Run independent tasks concurrently
  5. Incremental builds: Only rebuild what changed
  6. Dependency graph: Track package relationships
  7. Force full build: On main branch merges
  1. 拉取完整历史:设置
    fetch-depth: 0
    以获取准确的差异对比
  2. 拓扑顺序:优先构建依赖项
  3. 远程缓存:在CI运行之间共享缓存
  4. 并行执行:同时运行独立任务
  5. 增量构建:仅重新构建变更的内容
  6. 依赖图谱:跟踪包之间的依赖关系
  7. 强制完整构建:在主分支合并时执行

Output Checklist

输出检查清单

  • Affected detection implemented
  • Turborepo or Nx configured
  • Remote caching enabled
  • Parallel execution optimized
  • Matrix strategy for packages
  • Workspace-aware caching
  • Dependency graph documented
  • Before/after metrics tracked
  • 已实现受影响代码检测
  • 已配置Turborepo或Nx
  • 已启用远程缓存
  • 已优化并行执行
  • 已针对包配置矩阵策略
  • 已启用工作区感知缓存
  • 已记录依赖图谱
  • 已跟踪优化前后的指标