Compare Test Runs

Quick Start

1. Run

   tuist test show <id> --json

   for both base and head test runs.
2. Run

   tuist test module list <test-run-id> --json

   and

   tuist test suite list <test-run-id> --json

   to get module and suite breakdowns.
3. Run

   tuist test case run list <identifier> --json

   to get individual test case results.
4. Compare failures, flaky tests, durations, and overall status.
5. Inspect failing test cases with

   tuist test case run show <id> --json

   .
6. Summarize findings with actionable recommendations.

Step 1: Resolve Test Runs

If base/head are test run IDs or dashboard URLs

tuist test show <base-id> --json
tuist test show <head-id> --json

If base/head are branch names

tuist test list --git-branch <base-branch> --json --page-size 5
tuist test list --git-branch <head-branch> --json --page-size 5

Defaults

• If no base is provided, use the project's default branch (usually

  main

  ).
• If no head is provided, detect the current git branch.

Step 2: Compare Top-Level Metrics

status

duration

total_test_count

failed_test_count

flaky_test_count

avg_test_duration

Step 3: Get Module and Suite Breakdowns

tuist test module list <base-test-run-id> --json
tuist test module list <head-test-run-id> --json

tuist test suite list <base-test-run-id> --json
tuist test suite list <head-test-run-id> --json

Step 4: Get Individual Test Case Results

tuist test case run list <identifier> --json --page-size 100

name

module_name

suite_name

Step 5: Classify Changes

1. New failures: Tests that passed in base but failed in head.
2. Fixed tests: Tests that failed in base but passed in head.
3. Newly flaky: Tests not flaky in base but flaky in head.
4. No longer flaky: Tests that were flaky in base but stable in head.
5. New tests: Tests present in head but not in base.
6. Removed tests: Tests present in base but not in head.
7. Duration regressions: Tests with >50% duration increase.

Step 6: Inspect Failures

tuist test case run show <test-case-run-id> --json

• failures[].message

  -- the assertion or error message

• failures[].path

  -- source file path

• failures[].line_number

  -- exact line of failure

• failures[].issue_type

  -- type of issue

• repetitions

  -- if present, shows retry behavior (flaky detection)

• crash_report

  -- crash data if test runner crashed

Step 7: Inspect Attachments

tuist test case run show

• Screenshots or UI test artifacts
• Log files or crash reports
• Any diagnostic data attached to failing runs

Summary Format

1. Overall verdict: Better, worse, or neutral compared to base.
2. New failures: List each with failure message, file path, and line number.
3. New flaky tests: List with flakiness context.
4. Fixed tests: List tests that are now passing.
5. Duration: Overall and notable per-test regressions.
6. Recommendations: Actionable next steps for each issue.

Test Run Comparison: base (run-123 on main) vs head (run-456 on feature-x)

Status: success -> failure -- REGRESSION
Duration: 120.5s -> 145.2s (+21%)
Tests: 342 -> 345 (3 new tests)
Failures: 0 -> 2 (2 new failures)
Flaky: 1 -> 3 (2 newly flaky)

New Failures:
1. AuthModuleTests/LoginTests/test_login_with_expired_token
   Message: "Expected status 401, got 500"
   File: Tests/AuthModuleTests/LoginTests.swift:42
   Likely cause: Server error handling changed for expired tokens

2. NetworkTests/RetryTests/test_retry_on_timeout
   Message: "Timed out waiting for retry"
   File: Tests/NetworkTests/RetryTests.swift:87
   Likely cause: Timeout threshold too low after network layer refactor

Newly Flaky:
1. CacheTests/WriteCacheTests/test_concurrent_writes (flaky in 3/5 runs)

Recommendations:
- Fix expired token handling in AuthModule
- Increase timeout in RetryTests or mock the network layer
- Investigate concurrent write synchronization in CacheTests

Done Checklist

• Resolved both base and head test runs
• Compared top-level metrics
• Fetched module and suite breakdowns for both runs
• Identified new failures, fixed tests, and flaky changes
• Inspected failure details for new failures
• Provided actionable recommendations with file paths