fix-flaky-tests

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Fix Flaky Tests

修复不稳定测试

Quick Start

快速开始

You'll typically receive a Tuist test case URL or identifier. Follow these steps to investigate and fix it:

Run

tuist test case show <id-or-identifier> --json

to get reliability metrics for the test.

Run

tuist test case run list Module/Suite/TestCase --flaky --json

to see flaky run patterns.

Run
```
tuist test case run show <run-id> --json
```
on failing flaky runs to get failure messages and file paths.
Read the test source at the reported path and line, identify the flaky pattern, and fix it.
Verify by running the test multiple times to confirm it passes consistently.

If no specific test is provided, start with the Discovery section below.

你通常会收到一个Tuist测试用例URL或标识符。按照以下步骤调查并修复问题：

运行

tuist test case show <id-or-identifier> --json

获取该测试的可靠性指标。

运行

tuist test case run list Module/Suite/TestCase --flaky --json

查看不稳定运行模式。

针对失败的不稳定运行，运行
```
tuist test case run show <run-id> --json
```
获取失败信息和文件路径。
查看报告路径和行号对应的测试源码，识别不稳定模式并修复问题。
通过多次运行测试验证，确认其能持续通过。

如果未提供特定测试，请从下方的“发现”部分开始。

Discovery

发现

When no specific test case is provided, find all flaky tests in the project:

bash

tuist test case list --flaky --json --page-size 50

This returns all test cases currently flagged as flaky. Key fields:

```
module.name
```
/
```
suite.name
```
/
```
name
```
— the test identifier
```
avg_duration
```
— helps prioritize (fix fast unit tests first)
```
is_quarantined
```
— whether the test is already quarantined

Triage strategy:

Group tests by suite — multiple flaky tests in the same suite often share a root cause.
Check if failures share a
```
test_run_id
```
— tests that all failed in the same run may have been killed by a process crash, not individual test bugs.
Look at failure messages to categorize: test logic bugs vs infrastructure issues (network errors, server 502s, conflicts on retry).

当未提供特定测试用例时，查找项目中所有不稳定测试：

bash

tuist test case list --flaky --json --page-size 50

该命令返回所有当前标记为不稳定的测试用例。关键字段：

```
module.name
```
/
```
suite.name
```
/
```
name
```
—— 测试标识符
```
avg_duration
```
—— 帮助确定优先级（优先修复快速单元测试）
```
is_quarantined
```
—— 测试是否已被隔离

分类策略：

按测试套件分组 —— 同一套件中的多个不稳定测试通常共享同一根本原因。
检查失败是否共享
```
test_run_id
```
—— 在同一运行中全部失败的测试可能是被进程崩溃影响，而非单个测试的bug。
查看失败信息进行分类：测试逻辑bug vs 基础设施问题（网络错误、服务器502、重试冲突）。

Investigation

调查

1. Get test case metrics

1. 获取测试用例指标

You can pass either the UUID or the

Module/Suite/TestCase

identifier:

bash

tuist test case show <id> --json
tuist test case show Module/Suite/TestCase --json

Key fields:

```
reliability_rate
```
— percentage of successful runs (higher is better)
```
flakiness_rate
```
— percentage of runs marked flaky in the last 30 days
```
total_runs
```
/
```
failed_runs
```
— volume context
```
last_status
```
— current state

你可以传入UUID或

Module/Suite/TestCase

标识符：

bash

tuist test case show <id> --json
tuist test case show Module/Suite/TestCase --json

关键字段：

```
reliability_rate
```
—— 成功运行的百分比（越高越好）
```
flakiness_rate
```
—— 过去30天内标记为不稳定的运行百分比
```
total_runs
```
/
```
failed_runs
```
—— 运行量上下文
```
last_status
```
—— 当前状态

2. View flaky run history

2. 查看不稳定运行历史

bash

tuist test case run list Module/Suite/TestCase --flaky --json

The identifier uses the format

ModuleName/SuiteName/TestCaseName

ModuleName/TestCaseName

when there is no suite. This returns only runs that were detected as flaky.

bash

tuist test case run list Module/Suite/TestCase --flaky --json

标识符格式为

ModuleName/SuiteName/TestCaseName

，当没有套件时为

ModuleName/TestCaseName

。该命令仅返回被检测为不稳定的运行记录。

3. View full run history

3. 查看完整运行历史

bash

tuist test case run list Module/Suite/TestCase --json --page-size 20

Look for patterns:

Does it fail on specific branches?
Does it fail only on CI (
```
is_ci: true
```
) or also locally?
Are failures clustered around specific commits?

bash

tuist test case run list Module/Suite/TestCase --json --page-size 20

查找模式：

是否在特定分支上失败？
是否仅在CI环境（
```
is_ci: true
```
）失败，还是本地也会失败？
失败是否集中在特定提交附近？

4. Get failure details

4. 获取失败详情

bash

tuist test case run show <run-id> --json

Key fields:

```
failures[].message
```
— the assertion or error message
```
failures[].path
```
— source file path
```
failures[].line_number
```
— exact line of failure
```
failures[].issue_type
```
— type of issue (assertion_failure, etc.)
```
repetitions
```
— if present, shows retry behavior (pass/fail sequence)
```
test_run_id
```
— the broader test run this execution belongs to

bash

tuist test case run show <run-id> --json

关键字段：

```
failures[].message
```
—— 断言或错误信息
```
failures[].path
```
—— 源码文件路径
```
failures[].line_number
```
—— 失败的具体行号
```
failures[].issue_type
```
—— 问题类型（如assertion_failure）
```
repetitions
```
—— 若存在，显示重试行为（通过/失败序列）
```
test_run_id
```
—— 该执行所属的更广泛测试运行

Code Analysis

代码分析

Open the file at
```
failures[0].path
```
and go to
```
failures[0].line_number
```
.
Read the full test function and its setup/teardown.
Identify which of the common flaky patterns below applies.
Check if the test shares state with other tests in the same suite.

打开
```
failures[0].path
```
对应的文件，跳转到
```
failures[0].line_number
```
行。
阅读完整的测试函数及其setup/teardown代码。
识别符合以下哪种常见不稳定模式。
检查该测试是否与同一套件中的其他测试共享状态。

Common Flaky Patterns

常见不稳定模式

Timing and async issues

时序与异步问题

Missing waits: Test checks a result before an async operation completes. Fix: use
```
await
```
, expectations with timeouts, or polling.
Race conditions: Multiple concurrent operations access shared state. Fix: synchronize access or use serial queues.
Hardcoded timeouts:
```
sleep(1)
```
or fixed delays that are too short on CI. Fix: use condition-based waits instead of fixed delays.

缺失等待：测试在异步操作完成前检查结果。修复方案：使用
```
await
```
、带超时的预期或轮询。
竞态条件：多个并发操作访问共享状态。修复方案：同步访问或使用串行队列。
硬编码超时：
```
sleep(1)
```
或固定延迟在CI环境中过短。修复方案：使用基于条件的等待而非固定延迟。

Shared state

共享状态

Test pollution: One test modifies global/static state that another test depends on. Fix: reset state in setUp/tearDown or use unique instances per test.
Singleton contamination: Shared singletons carry state between tests. Fix: inject dependencies or reset singletons.
File system leftovers: Tests leave files that affect subsequent runs. Fix: use temporary directories and clean up.

测试污染：一个测试修改了全局/静态状态，而另一个测试依赖该状态。修复方案：在setUp/tearDown中重置状态，或为每个测试使用唯一实例。
单例污染：共享单例在测试间携带状态。修复方案：注入依赖或重置单例。
文件系统残留：测试留下的文件影响后续运行。修复方案：使用临时目录并清理。

Environment dependencies

环境依赖

Network calls: Tests hit real services that may be slow or unavailable. Fix: mock network calls.
Date/time sensitivity: Tests depend on current time or timezone. Fix: inject a clock or freeze time.
File system paths: Hardcoded paths that differ between environments. Fix: use relative paths or temp directories.

网络调用：测试访问真实服务，可能存在缓慢或不可用的情况。修复方案：模拟网络调用。
日期/时间敏感：测试依赖当前时间或时区。修复方案：注入时钟或冻结时间。
文件系统路径：硬编码路径在不同环境中存在差异。修复方案：使用相对路径或临时目录。

Order dependence

顺序依赖

Implicit ordering: Test passes only when run after another test that sets up required state. Fix: make each test self-contained.
Parallel execution conflicts: Tests that work in isolation but fail when run concurrently. Fix: use unique resources per test.

隐式顺序：测试仅在另一个设置了所需状态的测试之后运行时才能通过。修复方案：让每个测试独立自包含。
并行执行冲突：测试在隔离运行时正常，但并发运行时失败。修复方案：为每个测试使用唯一资源。

Fix Implementation

修复实施

After identifying the pattern:

Apply the smallest fix that addresses the root cause.
Do not refactor unrelated code.
If the fix requires a test utility (like a mock or helper), check if one already exists before creating a new one.

识别模式后：

应用最小的修复来解决根本原因。
不要重构无关代码。
如果修复需要测试工具（如mock或助手），先检查是否已存在，再创建新工具。

Verification

验证

Running tests repeatedly

重复运行测试

Run the specific test repeatedly until failure using

xcodebuild

's built-in repetition support:

bash

xcodebuild test -workspace <workspace> -scheme <scheme> -only-testing <module>/<suite>/<test> -test-iterations <count> -run-tests-until-failure

This runs the test up to

<count>

times and stops at the first failure. Choose the iteration count based on how long the test takes — for fast unit tests use 50–100, for slower integration or acceptance tests use 2–5.

使用

xcodebuild

内置的重复支持，重复运行特定测试直到失败：

bash

xcodebuild test -workspace <workspace> -scheme <scheme> -only-testing <module>/<suite>/<test> -test-iterations <count> -run-tests-until-failure

该命令最多运行测试

<count>

次，首次失败时停止。根据测试耗时选择迭代次数——快速单元测试使用50–100次，较慢的集成或验收测试使用2–5次。

Reproducing before fixing

修复前复现问题

Before applying a fix, try to reproduce the flaky failure locally. A successful reproduction confirms your root cause analysis and lets you verify the fix directly. Use the "Running tests repeatedly" approach above, or the race condition strategies below if concurrency is suspected.

Some flaky scenarios — especially race conditions, CI-specific timing issues, or environment-dependent failures — may be difficult or impossible to reproduce locally. If you cannot reproduce after reasonable effort, proceed with fixing based on code analysis and failure logs. A fix backed by clear evidence of a bug (e.g. unsynchronized shared state, TOCTOU pattern) is valid even without local reproduction.

在应用修复前，尝试在本地复现不稳定失败。成功复现可确认你的根本原因分析，并让你直接验证修复效果。使用上述“重复运行测试”的方法，或如果怀疑是竞态条件，使用以下策略。

一些不稳定场景——尤其是竞态条件、CI特定时序问题或环境依赖失败——可能难以或无法在本地复现。如果经过合理尝试仍无法复现，可基于代码分析和失败日志进行修复。有明确bug证据（如未同步的共享状态、TOCTOU模式）的修复即使没有本地复现也是有效的。

Reproducing race conditions

复现竞态条件

Race conditions and concurrency bugs often only manifest under CI-level parallelism and are hard to reproduce locally. Try these strategies in order:

Increase parallelism: Add
```
-parallel-testing-enabled YES
```
to run test suites concurrently.
Run broader test suites: Instead of running a single test, run the entire module (e.g.
```
-only-testing ModuleTests
```
) to increase contention on shared resources.
Thread Sanitizer: Run with TSan enabled to detect data races deterministically. Note: TSan adds overhead which can change timing, so some races may not trigger under TSan.

bash

xcodebuild test -workspace <workspace> -scheme <scheme> -only-testing <module> -enableThreadSanitizer YES

High iteration count with broad scope: Combine all the above — run the full module with parallelism and many iterations.

If a race condition cannot be reproduced locally but the code is provably thread-unsafe (e.g. unsynchronized mutation of shared state), the fix is still valid. Verify the fix by confirming the tests pass with the same reproduction strategies above. Document in the commit message that the fix addresses a CI-only race condition identified through code analysis and failure logs.

竞态条件和并发bug通常仅在CI级别的并行度下出现，难以在本地复现。按以下顺序尝试这些策略：

提高并行度：添加
```
-parallel-testing-enabled YES
```
以并发运行测试套件。
运行更广泛的测试套件：不要只运行单个测试，运行整个模块（例如
```
-only-testing ModuleTests
```
）以增加共享资源的竞争。
线程 sanitizer：启用TSan以确定性地检测数据竞争。注意：TSan会增加开销，可能改变时序，因此某些竞争可能不会在TSan下触发。

bash

xcodebuild test -workspace <workspace> -scheme <scheme> -only-testing <module> -enableThreadSanitizer YES

大范围高迭代次数：结合以上所有方法——以并行模式运行整个模块并执行多次迭代。

如果竞态条件无法在本地复现，但代码明显存在线程不安全问题（如未同步的共享状态修改），修复仍然有效。通过确认测试在上述复现策略下通过来验证修复。在提交信息中记录该修复解决了通过代码分析和失败日志识别的仅CI环境存在的竞态条件。

Done Checklist

完成检查清单

Identified the root cause of flakiness
Applied a targeted fix
Verified the test passes consistently (multiple runs)
Did not introduce new test dependencies or shared state
Committed the fix with a descriptive message

已识别不稳定的根本原因
已应用针对性修复
已验证测试能持续通过（多次运行）
未引入新的测试依赖或共享状态
已提交修复并附带描述性信息