selector-drift-recovery
Compare original and translation side by side
🇺🇸
Original
English🇨🇳
Translation
Chinese<objective>
A redesign shipped and 23 tests now fail with `TimeoutError: locator.* exceeded` — the DOM moved, not the product. This skill closes that maintenance loop: it diffs the old DOM against the new one, regenerates the broken selectors role-first against the new build, validates the whole suite, and ships the diff as one reviewable PR with a confidence score and screenshot per change. The trigger is an event (a refactor merged), not a flake. The output is a PR a human signs off on — not a silent runtime auto-heal.
This is the bulk, offline, batch counterpart to . They share the multi-attribute locator + confidence scoring primitives but run in opposite directions: heals one selector at runtime behind a guarded threshold; regenerates N selectors offline against the new DOM and bundles them into a PR. If you are doing the second workflow inside , switch here.
</objective>
test-reliabilitytest-reliabilityselector-drift-recoverytest-reliability<objective>
重新设计上线后,有23个测试因`TimeoutError: locator.* exceeded`失败——问题出在DOM结构变更,而非产品功能。本技能可完成这一维护闭环:对比新旧DOM结构,针对新版本优先按角色重新生成失效选择器,验证整个测试套件,并将差异打包为一个可评审的PR,每项变更附带置信度评分和截图。触发条件是事件(重构已合并),而非测试不稳定。输出结果是需要人工确认的PR,而非静默的运行时自动修复。
这是的批量离线版本。二者共享多属性定位器+置信度评分核心逻辑,但运行方向相反:在运行时针对单个选择器进行阈值管控下的修复;则离线针对新DOM重新生成N个选择器并打包为PR。如果你在中执行第二种工作流,请切换到本技能。
</objective>
test-reliabilitytest-reliabilityselector-drift-recoverytest-reliabilityQuick Route
快速指引
| Situation | Go to |
|---|---|
| One broken test, not a refactor | Stop — use |
| 200+ broken selectors across many files | Split by area first (one PR per page/dir), then Phase 1 |
| Refactor changed flows/semantics, not just structure | Stop — rewrite from specs with |
| Framework switch (Selenium → Playwright) | Stop — use |
| No old-DOM reference exists anywhere | Capture one (Phase 1) or scope down — without it this is "rewrite tests" |
| Have old + new DOM, ready to map | Phase 1 → 6 below |
| 场景 | 对应技能 |
|---|---|
| 单个测试失效,非重构导致 | 停止——改用 |
| 跨多个文件存在200+个失效选择器 | 先按区域拆分(每个页面/目录一个PR),再执行第一阶段 |
| 重构变更了流程/语义,而非仅结构 | 停止——使用 |
| 框架切换(Selenium → Playwright) | 停止——使用 |
| 无旧DOM参考 | 先捕获旧DOM(第一阶段)或缩小范围——无旧DOM时等同于「重写测试」 |
| 已有新旧DOM,准备映射 | 执行以下第一至第六阶段 |
Discovery Questions
调研问题
Check first — if it exists, use it and skip anything answered there. It identifies your E2E framework, selector strategy, and known fragile areas. Then:
.agents/qa-project-context.md- What triggered the drift? A planned refactor (Storybook can show the new DOM before merge), a shipped redesign (new DOM is in main), a dependency upgrade, or a Tailwind/CSS migration? The trigger decides whether you run pre-emptively or react to CI failures.
- What is the blast radius? A single component, a page, or the whole app? Single component: scope recovery to the test files that touch it. Global: budget half a day to a day, and decide which tests should be rewritten rather than re-selected.
- What is your current selector strategy? If selectors are mostly and the refactor preserved testids, recovery is trivial. If they are CSS-class or XPath based, expect 30–60% to need a new strategy, not just a new locator.
data-testid - Is there a passing baseline? You need the old DOM somewhere: a previous CI trace artifact, a deployed staging build, a Storybook story, or git history of the components. No old-DOM reference means this degrades to "rewrite tests."
- Are the broken locators inline or wrapped in a Page Object? The JSON reporter's points at the failing line. For inline locators that is the locator itself; for POM-wrapped locators it points at the POM helper, not the test. Know this before you trust the auto-extracted line numbers (see Failure Modes).
error.location - Who reviews the resulting PR? Confidence-scored updates need a human signoff. Decide upfront whether the PR goes to the test author, the engineer who did the refactor, or the QA lead.
首先查看——如果存在,使用其中信息并跳过已回答的问题。该文件会指明你的E2E框架、选择器策略及已知脆弱区域。然后:
.agents/qa-project-context.md- 是什么触发了漂移? 计划内重构(Storybook可在合并前展示新DOM)、已上线的重新设计(新DOM在主分支)、依赖升级,还是Tailwind/CSS迁移?触发方式决定了你是提前执行还是响应CI失败。
- 影响范围多大? 单个组件、单个页面,还是整个应用?单个组件:将修复范围限定在涉及该组件的测试文件;全局范围:预留半天到一天时间,并决定哪些测试应重写而非重新选择。
- 当前使用的选择器策略是什么? 如果选择器大多是且重构保留了testid,修复会非常简单;如果是基于CSS类或XPath的选择器,预计30%-60%需要更换策略,而非仅更换定位器。
data-testid - 是否有可用的通过基线? 你需要在某处获取旧DOM:之前的CI跟踪 artifact、已部署的预发布版本、Storybook示例,或组件的git历史。无旧DOM参考时,本技能退化为「重写测试」。
- 失效的定位器是内联还是封装在Page Object中? JSON报告的指向失败行。对于内联定位器,该行就是定位器本身;对于POM封装的定位器,该行指向POM助手而非测试代码。在信任自动提取的行号前需明确这一点(参见故障模式)。
error.location - 谁来评审最终PR? 带置信度评分的更新需要人工确认。提前确定PR提交给测试作者、执行重构的工程师还是QA负责人。
Core Principles
核心原则
-
Recovery is event-driven, not failure-driven. Run this when a refactor is planned or just shipped, not when one test goes flaky. One broken test →. Ten or more from the same event → this skill.
test-reliability -
Old DOM, new DOM, mapped pair. The whole skill rests on a snapshot before the refactor and one after. Everything else is bookkeeping around that pair. Capture both as aria snapshots (), not raw HTML — a role-tree diff is exactly the signal role-first recovery needs and ignores the cosmetic churn (class renames, wrapper divs) that a raw-HTML diff drowns in. If you cannot produce both snapshots, fix that first.
await page.locator('body').ariaSnapshot() -
Role-first replacement, every time. Even when the old test used CSS, the regenerated selector should prefer+ accessible name, then
getByRolefor form fields, thengetByLabelwhen the refactor added one. The recovery PR is your chance to ratchet the average selector stability score up (0–5 rubric below, shared withgetByTestId).test-reliability -
Disambiguate by region scoping, not layout selectors. When role+name is ambiguous (two "Submit" buttons), narrow withor
getByRole('region', { name }).getByRole('button', …). Do not reach forgetByRole(...).filter({ hasText })/:near()— see the Avoid note. A score-3 candidate is one where region scoping has been applied and the locator now matches exactly one element.:right-of() -
One PR, grouped by file, with per-change evidence. Reviewers cannot eyeball 47 selector changes spread across 30 commits. Bundle into one PR, group hunks by test file, attach a confidence score + DOM screenshot per change.
-
The suite must pass before merge, and dead tests get deleted. A regenerated selector that doesn't run is a worse version of the original problem; the skill ends with green CI, not a generated diff. And if the refactor removed a feature, prune its tests — do not regenerate selectors for elements that no longer exist.
Avoid: Playwright layout selectors,:near(),:right-of(),:left-of(),:above()as disambiguators — officially deprecated and "may be removed," because a 1px layout shift changes the match (Playwright docs, 2026). They also contradict the role-first thesis. Use region scoping /:below()instead.getByRole().filter()
-
修复由事件驱动,而非失败驱动。 在计划重构或重构刚完成时执行本技能,而非单个测试不稳定时。单个失效测试→使用;同一事件导致10个及以上测试失效→使用本技能。
test-reliability -
旧DOM、新DOM、映射配对。 本技能的核心依赖重构前后的两个快照。所有其他操作都是围绕这对快照的辅助工作。将两者捕获为aria快照(),而非原始HTML——角色树对比正是优先按角色修复所需的信号,且会忽略原始HTML对比中会干扰判断的 cosmetic 变更(类名重命名、包装div等)。如果无法生成两个快照,请先解决此问题。
await page.locator('body').ariaSnapshot() -
始终优先按角色替换。 即使旧测试使用CSS选择器,重新生成的选择器也应优先使用+可访问名称,然后是表单字段的
getByRole,最后是重构新增的getByLabel。修复PR是提升选择器平均稳定性评分的机会(评分标准为0-5,与getByTestId共享)。test-reliability -
通过区域范围消除歧义,而非布局选择器。 当角色+名称存在歧义时(如两个“Submit”按钮),使用或
getByRole('region', { name }).getByRole('button', …)缩小范围。请勿使用getByRole(...).filter({ hasText })/:near()——参见注意事项。评分3的候选选择器是指已应用区域范围且定位器仅匹配一个元素的情况。:right-of() -
单个PR,按文件分组,附带每项变更证据。 评审人员无法逐一查看分散在30个提交中的47项选择器变更。将所有变更打包为一个PR,按测试文件分组,每项变更附带置信度评分+DOM截图。
-
合并前套件必须通过测试,失效测试需删除。 无法运行的重新生成选择器比原问题更糟;本技能的终点是CI通过,而非生成差异。如果重构移除了某个功能,需删除对应测试——不要为已不存在的元素重新生成选择器。
注意事项: 避免使用Playwright布局选择器、:near()、:right-of()、:left-of()、:above()作为歧义消除方式——官方已标记为弃用,“可能会被移除”,因为1px的布局偏移就会改变匹配结果(Playwright文档,2026)。同时这也与优先按角色的原则相悖,请改用区域范围划分/:below()。getByRole().filter()
Workflow
工作流程
Six phases, each gated by a check before the next.
分为六个阶段,每个阶段完成后需通过检查才能进入下一阶段。
Phase 1 — Snapshot the old DOM
阶段1 — 快照旧DOM
You need a snapshot of every page or component the affected tests touch, in its pre-refactor state. Sources, in preference order:
- Last green CI trace artifact. Most teams save Playwright traces on failure (). Download the green-run artifact, open a trace with
trace: 'on-first-retry', select an action, and read the per-action DOM snapshot panel for each surface. (The viewer no longer has a "Copy HTML at this step" menu item; you read the snapshot panel or, for a programmatic dump, replay withnpx playwright show-trace traces/checkout.zip/page.content().)ariaSnapshot() - Storybook at a pre-refactor commit. , start Storybook, and dump each story with a tiny
git checkout <PRE_REFACTOR_SHA>/page.content()script.ariaSnapshot() - A staging build still on the old version. Navigate the same flows and snapshot.
- Git history of the components. Reconstructable but the most expensive — render in isolation.
Output: (and if you also need raw markup) per affected unit.
.drift-recovery/old/<page-or-component>.aria.yml.htmlGate: You can answer "what did this page look like when the tests last passed?" from a snapshot file, not from memory.
你需要获取受影响测试涉及的每个页面或组件在重构前的状态快照。来源优先级:
- 最后一次通过的CI跟踪artifact。 大多数团队会在测试失败时保存Playwright跟踪()。下载通过版本的artifact,使用
trace: 'on-first-retry'打开跟踪记录,选择操作步骤,查看每个界面的DOM快照面板。(查看器不再有“复制此步骤的HTML”菜单项;你可以读取快照面板,或通过npx playwright show-trace traces/checkout.zip/page.content()以编程方式导出。)ariaSnapshot() - 重构前提交的Storybook。 执行,启动Storybook,使用小型
git checkout <PRE_REFACTOR_SHA>/page.content()脚本导出每个示例。ariaSnapshot() - 仍为旧版本的预发布构建。 导航相同流程并生成快照。
- 组件的git历史。 可重构但成本最高——需单独渲染。
输出:每个受影响单元对应(若需原始标记可同时生成文件)。
.drift-recovery/old/<page-or-component>.aria.yml.html检查点: 你可以通过快照文件而非记忆回答“测试最后通过时该页面是什么样子?”
Phase 2 — Snapshot the new DOM
阶段2 — 快照新DOM
Run the same surfaces in the post-refactor build — a Vercel/Netlify preview deploy is ideal, or a local dev server / the PR branch in CI. Wait for hydration () before snapshotting, or SSR pages give you the pre-hydration tree and you miss client-rendered elements.
await page.waitForLoadState('networkidle')Output: matching the old set.
.drift-recovery/new/<page-or-component>.aria.ymlGate: Every old snapshot has a matching new one. If a route 404s in the new build, that flow was deleted — mark its tests for the deletion pile in Phase 6.
在重构后的版本中运行相同界面——Vercel/Netlify预览部署是理想选择,也可使用本地开发服务器/CI中的PR分支。在生成快照前等待 hydration 完成(),否则SSR页面会返回预hydration树,导致缺失客户端渲染元素。
await page.waitForLoadState('networkidle')输出:与旧快照对应的文件。
.drift-recovery/new/<page-or-component>.aria.yml检查点: 每个旧快照都有对应的新快照。如果新构建中某路由返回404,说明该流程已被删除——在阶段6标记对应测试为待删除。
Phase 3 — Identify broken selectors and infer intent
阶段3 — 识别失效选择器并推断意图
For each test file, run against the new build with the JSON reporter, then parse it. Capture, per failure: file, line, old locator string, error type (timeout vs assertion), and inferred intent. Group the results by test file.
- Error classification: a drift failure is . Distinguish it from an assertion failure (
TimeoutError: locator.* exceeded) so you don't try to re-select a locator that resolved fine but failed a value check.expect(...).toBe - Inferred intent is mandatory and not in the reporter. Read the surrounding test code — what action is taken on the locator, what assertion follows — and store a short intent string ("submit the order", "read the order total"). The candidate generator keys off intent, so this step is load-bearing, not commentary.
- Page route is also not in the reporter. Map each locator to the snapshot it should resolve against (which ) so Phase 4 can load the right new DOM.
.drift-recovery/new/*.aria.yml
The result is a per-file table:
| Test file | Line | Old locator | Error type | Page route | Inferred intent |
|---|---|---|---|---|---|
| 42 | | timeout | | Submit the order |
| 87 | | timeout | | Read the order total |
See for , which produces exactly these rows (with the intent/route fields populated, not stubbed).
references/recovery-scripts.mdidentify-drift.tsGate: Every broken locator has an inferred intent and a page route. If you cannot infer intent, ask the test author or read the original PR — do not guess.
针对每个测试文件,在新版本上运行并使用JSON报告器,然后解析报告。捕获每项失败的文件、行号、旧定位器字符串、错误类型(超时vs断言)及推断意图。按测试文件分组结果。
- 错误分类: 漂移失败为。需与断言失败(
TimeoutError: locator.* exceeded)区分,避免尝试重新选择已正常解析但值检查失败的定位器。expect(...).toBe - 推断意图是必需的,且报告中不包含该信息。 阅读测试代码上下文——定位器执行的操作是什么,后续的断言是什么——并存储简短的意图字符串(如“提交订单”、“读取订单总额”)。候选选择器生成器依赖此意图,因此这一步是核心,而非注释。
- 页面路由也不在报告中。 将每个定位器映射到其应解析的快照(即),以便阶段4加载正确的新DOM。
.drift-recovery/new/*.aria.yml
结果为按文件分组的表格:
| 测试文件 | 行号 | 旧定位器 | 错误类型 | 页面路由 | 推断意图 |
|---|---|---|---|---|---|
| 42 | | 超时 | | 提交订单 |
| 87 | | 超时 | | 读取订单总额 |
参见中的,该脚本可生成上述完整行(已填充意图/路由字段,而非占位符)。
references/recovery-scripts.mdidentify-drift.ts检查点: 每个失效定位器都有推断意图和页面路由。如果无法推断意图,请询问测试作者或查看原始PR——请勿猜测。
Phase 4 — Generate replacement candidates
阶段4 — 生成替换候选选择器
For each row, generate candidates against the new DOM snapshot and score each on the 0–5 rubric (shared with ). Strategy ladder, best first:
test-reliability- New added by the refactor team — the most stable choice they made. Score 5.
data-testid - + accessible name, unique on the page. Score 4.
getByRole - for a form field, when the intent is an input and a label exists. Score 4 (use over a bare role when the field has no name otherwise).
getByLabel - Role + name, region-scoped to a single match. If role+name alone returns >1 element, wrap it — or
getByRole('region', { name }).getByRole(role, { name })— and confirm the scoped locator now matches exactly one. Only score 3 after scoping makes it unambiguous..filter({ hasText }) - Visible text only (). Score 2 — fragile to copy changes.
getByText - CSS class on the changed structure. Score 1 — usually still broken.
- No safe replacement. Score 0 — flag for human.
| Score | Replacement strategy | Auto-apply? |
|---|---|---|
| 5 | New | yes |
| 4 | | yes |
| 3 | | yes |
| 2 | Visible-text-only | no |
| 1 | CSS class on changed structure | no |
| 0 | No safe replacement found | no — flag for human |
A candidate is score 3 only if scoping already resolved it to a single element. A still-ambiguous multi-match (, "needs scoping") is not a 3 — it is unfinished, and must not be auto-applied.
count > 1Output: with per change. See for .
.drift-recovery/candidates.json{ file, line, oldLocator, selector, score, rationale, screenshotPath }references/recovery-scripts.mdgenerate-candidates.tsGate: Every row has a candidate scored ≥ 3 (and confirmed unique), or is flagged for human review. Score-0/1/2 are never auto-applied.
针对每一行,在新DOM快照上生成候选选择器,并按0-5评分标准(与共享)评分。策略优先级从高到低:
test-reliability- 重构团队新增的——最稳定的选择。评分5。
data-testid - +可访问名称,在页面上唯一。 评分4。
getByRole - 表单字段的——当意图是输入且存在标签时使用。评分4(当字段无其他名称时,优先于单纯的角色选择器)。
getByLabel - 角色+名称,通过区域范围限定为单一匹配。 如果仅角色+名称返回>1个元素,需进行包装——或
getByRole('region', { name }).getByRole(role, { name })——并确认范围限定后的定位器仅匹配一个元素。仅在范围限定消除歧义后才可评为3分。.filter({ hasText }) - 仅可见文本()。评分2——易受文本变更影响。
getByText - 变更结构上的CSS类。 评分1——通常仍会失效。
- 无安全替换方案。 评分0——标记为需人工处理。
| 评分 | 替换策略 | 自动应用? |
|---|---|---|
| 5 | 存在新增的 | 是 |
| 4 | | 是 |
| 3 | | 是 |
| 2 | 仅可见文本 | 否 |
| 1 | 变更结构上的CSS类 | 否 |
| 0 | 未找到安全替换方案 | 否——标记为需人工评审 |
候选选择器仅在范围限定后解析为单一元素时才可评为3分。仍存在歧义的多匹配(,“需范围限定”)不能评为3分——属于未完成状态,不得自动应用。
count > 1输出:,每项变更包含。参见中的。
.drift-recovery/candidates.json{ file, line, oldLocator, selector, score, rationale, screenshotPath }references/recovery-scripts.mdgenerate-candidates.ts检查点: 每一行都有评分≥3的候选选择器(且已确认唯一),或已标记为需人工评审。评分0/1/2的候选选择器绝不自动应用。
Phase 5 — Apply, validate, iterate
阶段5 — 应用、验证、迭代
- Apply the score-≥3 replacements to a feature branch. Replace by , not a content-wide string replace — the reporter's locator string is a rendered form (
(file, line)) that rarely matches the source expression verbatim, and a naivelocator('.summary > h2')hits only the first occurrence and collides on identical locators. Edit the specific line; setString.replaceon each candidate you actually wrote.applied: true - Run the full affected suite, not just the previously-failing tests — a new selector can match an unintended element and break a previously-passing test.
- Per test: passed → keep the replacement. failed → revert that one line, mark the test for human review.
- Emit a summary: N recovered automatically, M flagged.
Gate: Recovered tests pass the suite. Flagged tests are clearly marked, not silently included.
- 将评分≥3的替换项应用到功能分支。按进行替换,而非全局字符串替换——报告中的定位器字符串是渲染后的形式(如
(file, line)),很少与源代码表达式完全匹配,简单的locator('.summary > h2')只会命中第一个匹配项,且会在相同定位器上发生冲突。编辑指定行;在实际应用的候选选择器上标记String.replace。applied: true - 运行所有受影响的测试套件,而非仅之前失败的测试——新选择器可能匹配非预期元素,导致之前通过的测试失效。
- 针对每个测试:通过→保留替换项;失败→还原该行,标记测试为需人工评审。
- 生成总结:自动修复N个,标记M个需人工处理。
检查点: 修复后的测试通过套件验证。标记的测试需明确标识,而非静默包含。
Phase 6 — Ship the PR
阶段6 — 提交PR
The PR is the deliverable. Title: . Body (generated from , filtering on ):
chore(tests): selector recovery after <refactor description>candidates.jsonappliedmarkdown
undefinedPR是最终交付物。标题: 。正文(从生成,过滤为true的项):
chore(tests): selector recovery after <refactor description>candidates.jsonappliedmarkdown
undefinedTrigger
触发原因
<Link to the refactor PR / describe the redesign>
<重构PR链接/描述重新设计内容>
Summary
总结
- N test files updated - M selectors changed
- K tests deleted (feature removed) - L tests flagged for manual review
- 更新N个测试文件 - 变更M个选择器
- 删除K个测试(功能已移除) - L个测试标记为需人工评审
Per-file changes
按文件变更
<For each file: a table of line, old, new, score, screenshot URL>
<针对每个文件:行号、旧定位器、新定位器、评分、截图URL的表格>
Flagged for review
需评审的测试
<Tests where no candidate scored >= 3, with the inferred intent>
<无评分≥3候选选择器的测试,附带推断意图>
How to review
评审指南
- Check each screenshot: does point at the element
newpointed at?old - For score-3 candidates, verify the region scope is meaningful in the new design.
- For flagged tests, decide: rewrite, delete, or accept a manual selector update.
Attach screenshots inline via your team's CI artifact URL pattern. See `references/recovery-scripts.md` for `apply-recovery.ts` (line-anchored) and `build-pr-body.ts`.
**Gate:** PR is reviewable in one sitting. Too large → split by area (one PR per page / component / test directory).
---- 查看每张截图:新定位器是否指向旧定位器对应的元素?
- 对于评分3的候选选择器,验证区域范围在新设计中是否合理。
- 对于标记的测试,决定:重写、删除或接受人工选择器更新。
通过团队的CI artifact URL模式将截图嵌入正文。参见`references/recovery-scripts.md`中的`apply-recovery.ts`(按行定位)和`build-pr-body.ts`。
**检查点:** PR可在一次评审中完成。若过大→按区域拆分(每个页面/组件/测试目录一个PR)。
---Anti-Patterns
反模式
-
Auto-applying score-0, -1, or -2 candidates. A score-2 is "we found some element." That is gambling, not recovery — and it is the usual cause of a suite-wide stability score dropping after a recovery. Auto-apply only score ≥ 3.
-
Calling an ambiguous multi-match "score 3." Ifreturns >1 element it is not a 3 until region scoping narrows it to exactly one. Scoring it 3 and auto-applying ships a locator that resolves to the wrong element.
getByRole(...) -
Skipping the screenshots. A score-4 candidate can still point at the wrong element when the page has two regions with the same role + name. The per-change screenshot is the only check that catches semantic drift; confidence scores alone do not.
-
Content-wide string replace instead of line-anchored edits.hits the first occurrence only, collides on duplicate locators, and silently no-ops when the reporter's rendered string differs from the source expression. Edit the specific
content.replace(oldLocator, …).(file, line) -
Trusting auto-extracted line numbers for POM-wrapped locators. The JSON reporter'spoints at the failing line, which for a Page Object is the helper, not the test. Re-read the locator from the trace action or grep the POM source before applying.
error.location -
Recovering tests for deleted features. The refactor may have removed flows. Map deleted routes in Phase 2 and prune those tests — do not regenerate selectors for elements that no longer exist.
-
Auto-merging the recovery PR in CI. The PR is the artifact; the whole point is a human eyeballing the per-change evidence. Auto-merge once tests pass defeats the purpose — a green suite with a selector pointing at the wrong-but-present element passes and erodes trust. Require a reviewer.
-
Treating the PR as urgent. A failed suite feels urgent; a correctly recovered one is what matters. Time pressure produces score-2 replacements that quietly degrade the suite.
-
自动应用评分0、1或2的候选选择器。 评分2意味着“我们找到了某个元素”,这是碰运气而非修复——也是修复后套件整体稳定性评分下降的常见原因。仅自动应用评分≥3的候选选择器。
-
将歧义多匹配标记为“评分3”。 如果返回>1个元素,需通过区域范围限定为单一匹配后才可评为3分。将其评为3分并自动应用会导致定位器解析到错误元素。
getByRole(...) -
跳过截图。 当页面存在两个相同角色+名称的区域时,评分4的候选选择器仍可能指向错误元素。每项变更的截图是捕获语义漂移的唯一检查方式;仅靠置信度评分无法做到。
-
全局字符串替换而非按行编辑。只会命中第一个匹配项,在重复定位器上发生冲突,且当报告中的渲染字符串与源代码表达式不同时会静默无操作。请编辑指定的
content.replace(oldLocator, …)。(file, line) -
信任POM封装定位器的自动提取行号。 JSON报告的指向失败行,对于Page Object来说是助手代码而非测试代码。应用前需从跟踪操作中读取定位器,或在POM源代码中搜索渲染后的字符串。
error.location -
为已删除功能的测试修复选择器。 重构可能已移除某些流程。在阶段2映射已删除的路由并删除对应测试——不要为已不存在的元素重新生成选择器。
-
在CI中自动合并修复PR。 PR本身就是交付物;核心目的是人工查看每项变更的证据。测试通过后自动合并会违背初衷——选择器指向错误但存在的元素时,套件会通过测试并逐渐降低信任度。需要求评审人员确认。
-
将PR视为紧急任务。 失效套件会让人感觉紧急,但正确修复才是关键。时间压力会导致应用评分2的替换项,悄悄降低套件质量。
Failure Modes
故障模式
| Symptom | Likely cause | Fix or check |
|---|---|---|
| Suite errored before producing the JSON report, or you parsed the wrong file | |
| Extracted line points at a POM file, not the test | Locator is wrapped in a Page Object | Read the locator from the trace action, or grep the POM source for the rendered string |
| Phase 3 output missing the inferred-intent / page-route fields | Populate them in Phase 3 — they are not in the reporter; the generator cannot infer them |
| New-DOM snapshot is missing client-rendered elements | Snapshotted before hydration | Add |
| Average stability score dropped after recovery | Score-2/CSS candidates auto-applied | Revert candidates with score < 3 in |
| 症状 | 可能原因 | 修复或检查方法 |
|---|---|---|
| 套件在生成JSON报告前出错,或解析了错误文件 | 执行 |
| 提取的行指向POM文件而非测试文件 | 定位器封装在Page Object中 | 从跟踪操作中读取定位器,或在POM源代码中搜索渲染后的字符串 |
| 阶段3输出缺失推断意图/页面路由字段 | 在阶段3填充这些字段——报告中不包含这些信息,生成器无法自动推断 |
| 新DOM快照缺失客户端渲染元素 | 快照在hydration前生成 | 在 |
| 修复后平均稳定性评分下降 | 自动应用了评分2/CSS候选选择器 | 还原 |
Verification
验证
Run these on the recovery branch before opening the PR, smallest first:
bash
undefined在打开PR前,在修复分支上按以下顺序执行验证:
bash
undefined1. No applied candidate is below the stability floor (machine-checkable proxy for "ratcheted up")
1. 所有应用的候选选择器均不低于稳定性阈值(可机器检查的“质量提升”指标)
node references/score-candidates.mjs .drift-recovery/candidates.json
node references/score-candidates.mjs .drift-recovery/candidates.json
prints average score + count of applied rows with score < 3 — that count MUST be 0
打印平均评分+评分<3的已应用项数量——该数量必须为0
2. The recovered suite is green against the new build
2. 修复后的套件在新版本上运行通过
PLAYWRIGHT_TEST_BASE_URL=$PREVIEW_URL npx playwright test --reporter=json
| jq '.stats.unexpected' # must be 0 (flagged tests excluded via grep/skip)
| jq '.stats.unexpected' # must be 0 (flagged tests excluded via grep/skip)
PLAYWRIGHT_TEST_BASE_URL=$PREVIEW_URL npx playwright test --reporter=json
| jq '.stats.unexpected' # 必须为0(标记的测试已通过grep/skip排除)
| jq '.stats.unexpected' # 必须为0(标记的测试已通过grep/skip排除)
3. The PR exists with the evidence body
3. PR已创建并包含证据正文
gh pr view --json title,body -q '.title' # contains "selector recovery"
`references/score-candidates.mjs` reads `candidates.json`, prints the average applied score and the count of `applied && score < 3` rows; a non-zero count means a low-confidence selector leaked in. Step 1 passing + step 2 returning `0` is the proof the recovery worked.
---gh pr view --json title,body -q '.title' # 包含“selector recovery”
`references/score-candidates.mjs`读取`candidates.json`,打印已应用项的平均评分及`applied && score < 3`的项数;非零数值意味着低置信度选择器被混入。步骤1通过+步骤2返回`0`是修复成功的证明。
---Done When
完成标准
- Every in-scope test passes on the new build, is flagged for human review with a clear reason, or is deleted because its feature is gone.
- reports 0 applied candidates with score < 3.
references/score-candidates.mjs candidates.json - The PR is open (succeeds) with per-change evidence: confidence score and screenshot per change, grouped by file.
gh pr view - returns
npx playwright test --reporter=json | jq '.stats.unexpected'on the recovery branch.0 - A short note is added to describing the refactor and any new test patterns introduced.
.agents/qa-project-context.md
- 所有范围内的测试在新版本上通过、被标记为需人工评审(附带明确原因),或因功能已移除被删除。
- 报告0个评分<3的已应用候选选择器。
references/score-candidates.mjs candidates.json - PR已创建(执行成功),并附带每项变更的证据:置信度评分和截图,按文件分组。
gh pr view - 在修复分支上执行返回
npx playwright test --reporter=json | jq '.stats.unexpected'。0 - 在中添加简短说明,描述重构内容及引入的新测试模式。
.agents/qa-project-context.md
Reference Files (in references/
)
references/参考文件(位于references/
)
references/- recovery-scripts.md — the full playbook with corrected, runnable scripts: aria-snapshot capture, (populates intent + route),
identify-drift.ts(region-scoping ladder, true score-3), line-anchoredgenerate-candidates.ts, andapply-recovery.ts. Includes the Cypress-adaptation note.build-pr-body.ts - score-candidates.mjs — tiny stability scorer; prints the average applied score and the count of applied rows below score 3. Used by Verification and Done When.
- recovery-scripts.md — 完整操作指南及可运行的修正脚本:aria快照捕获、(填充意图+路由)、
identify-drift.ts(区域范围优先级、真实评分3)、按行定位的generate-candidates.ts及apply-recovery.ts。包含Cypress适配说明。build-pr-body.ts - score-candidates.mjs — 小型稳定性评分器;打印已应用项的平均评分及评分<3的已应用项数。用于验证及完成标准检查。
Related Skills
相关技能
- test-reliability — runtime per-test healing. Use for one flaky test, not a refactor-driven mass update. Shares the 0–5 stability rubric.
- playwright-automation — writing new tests from scratch. Use when the refactor removed enough features that tests should be rewritten, not patched.
- test-migration — switching frameworks (Selenium → Playwright). A migration re-records tests; it is not selector drift, even though both touch many tests at once.
- visual-testing — had this run on every PR, the refactor's visual diff would have flagged before merge. Recovery is the fallback when that coverage is missing.
- ci-cd-integration — wires the recovery PR's validation step into CI.
- test-reliability — 运行时单测试修复。适用于单个不稳定测试,而非重构导致的批量更新。共享0-5稳定性评分标准。
- playwright-automation — 从头编写新测试。当重构移除大量功能,测试需重写而非修补时使用。
- test-migration — 框架切换(Selenium → Playwright)。迁移是重新录制测试;不属于选择器漂移,尽管二者都会同时修改大量测试。
- visual-testing — 如果在每个PR上运行该技能,重构的视觉差异会在合并前被标记。当该覆盖缺失时,修复是备选方案。
- ci-cd-integration — 将修复PR的验证步骤接入CI。