scrape-codegen-analyze

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

You are analyzing a detail page to produce extraction instructions for a code generation system. Given an HTML page, a schema, and expected values, you determine WHERE and HOW each field can be extracted from the page.

Read

${CLAUDE_SKILL_DIR}/../scrape/references/python-environments.md

Your analysis will be read by a separate code-generation agent that does not have access to the HTML. It must be detailed enough for that agent to write correct web-poet extraction code.

你正在分析一个详情页，为代码生成系统生成提取指令。给定一个HTML页面、一个schema和预期值，你需要确定每个字段可以从页面的何处以及如何提取。

请阅读

${CLAUDE_SKILL_DIR}/../scrape/references/python-environments.md

。

你的分析结果将由一个无法访问HTML的独立代码生成Agent读取。分析内容必须足够详细，以便该Agent编写正确的web-poet提取代码。

Input

输入

The raw argument string is

$ARGUMENTS

. Split it into 4 whitespace-separated positional arguments:

page_html_path: path to saved HTML file, e.g.
```
.scrape/spec/pages/detail-1/raw.html
```
work_path: working directory for saving analysis output, e.g.
```
.scrape/.work/spec
```
spec_path: path to spec.json file, e.g.
```
.scrape/spec/spec.json
```
values_path: path to values JSON file for this page, e.g.
```
.scrape/spec/values/detail-1.json
```

Plus, taken from the surrounding prompt text (not from the argument string):

fields: optional, specific fields to analyze (provided in the prompt as "Only analyze these fields: ..."). When set, only analyze those fields from the schema — skip the rest. When not set, analyze all fields.

The page directory (parent of the HTML file) also contains

meta.json

with the source URL.

原始参数字符串为

$ARGUMENTS

。将其拆分为4个以空格分隔的位置参数：

page_html_path：已保存HTML文件的路径，例如
```
.scrape/spec/pages/detail-1/raw.html
```
work_path：用于保存分析输出的工作目录，例如
```
.scrape/.work/spec
```
spec_path：spec.json文件的路径，例如
```
.scrape/spec/spec.json
```
values_path：此页面的预期值JSON文件路径，例如
```
.scrape/spec/values/detail-1.json
```

另外，从周围的提示文本中获取（而非来自参数字符串）：

fields：可选参数，指定需要分析的特定字段（在提示中以“仅分析以下字段：...”形式提供）。设置后，仅分析schema中的这些字段，跳过其余字段。未设置时，分析所有字段。

HTML文件的父目录中还包含带有源URL的

meta.json

文件。

Process

流程

1. Read inputs and prepare HTML

1. 读取输入并准备HTML

Derive the page_id from the directory name (e.g.

detail-1

from

.../detail-1/raw.html

Read

meta.json

from the same directory for the source URL and

page_type

meta.json

has

"page_type": "list"

, or the page_id starts with

list-

, set

is_list_page = true

. This changes how fields are analyzed (see step 2).

Read the schema from

{spec_path}

— use the

properties

object inside

schema

. Read the expected values from

{values_path}

— use the

values

object (may be

{}

Clean the HTML and extract structured metadata. Use level 0 cleaning to preserve scripts (which may contain JSON-LD/embedded data):

bash

mkdir -p {work_path}/codegen-analyze
uv run ${CLAUDE_SKILL_DIR}/../scrape-analyze-page/scripts/clean_html.py PAGE.html -l0 -o {work_path}/codegen-analyze/{page_id}.cleaned.html
uv run ${CLAUDE_SKILL_DIR}/../scrape-analyze-page/scripts/extract_metadata.py PAGE.html -u PAGE_URL -o {work_path}/codegen-analyze/{page_id}.metadata.json

Read only the cleaned HTML (not the original) and the metadata JSON.

从目录名称中派生page_id（例如从

.../detail-1/raw.html

中提取

detail-1

）。

从同一目录读取

meta.json

，获取源URL和

page_type

。

如果

meta.json

中包含

"page_type": "list"

，或者page_id以

list-

开头，则设置

is_list_page = true

。这会改变字段的分析方式（见步骤2）。

从

{spec_path}

读取schema — 使用

schema

内部的

properties

对象。从

{values_path}

读取预期值 — 使用

values

对象（可能为

{}

）。

清理HTML并提取结构化元数据。使用0级清理以保留脚本（其中可能包含JSON-LD/嵌入式数据）：

bash

mkdir -p {work_path}/codegen-analyze
uv run ${CLAUDE_SKILL_DIR}/../scrape-analyze-page/scripts/clean_html.py PAGE.html -l0 -o {work_path}/codegen-analyze/{page_id}.cleaned.html
uv run ${CLAUDE_SKILL_DIR}/../scrape-analyze-page/scripts/extract_metadata.py PAGE.html -u PAGE_URL -o {work_path}/codegen-analyze/{page_id}.metadata.json

仅读取清理后的HTML（而非原始HTML）和元数据JSON。

2. Analyze each field

2. 分析每个字段

For list pages (
is_list_page = true
): The page contains multiple repeated item elements. Instead of locating a single field value, you must identify:

The container selector — the CSS selector for the repeating element that wraps each item (e.g.,
```
article.product_pod
```
,
```
li.col-xs-6 article
```
,
```
div.item
```
). Look for the smallest repeated element that contains ALL requested fields. Report this as
```
container_selector
```
in the analysis output (see step 4).
Per-field selectors relative to the container — for each field, note the CSS selector that works inside a single container element (e.g., if the container is
```
article.product_pod
```
, the title might be
```
h3 a::attr(title)
```
relative to it). Verify that this relative selector works consistently across multiple container instances on the page.
Expected item count — note how many container elements you find; this should match the number of items in the values array.

The analysis for a list page should describe the container-based extraction pattern so

scrape-codegen-generate

can produce a

for container in self.css(...):

loop.

For detail pages: For each field in the schema, produce a detailed analysis. Consider all possible data sources in the HTML:

HTML elements: tags, classes, IDs, attributes — note the CSS selectors or XPaths
JSON-LD:
```
<script type="application/ld+json">
```
blocks — note the JSON path
Microdata:
```
itemscope
```
/
```
itemprop
```
attributes
OpenGraph:
```
<meta property="og:...">
```
tags
Other script tags: embedded JSON in
```
<script>
```
tags (e.g.
```
window.__DATA__ = {...}
```
)
URL components: data derivable from the page URL
Meta tags:
```
<meta name="...">
```
tags
Hidden inputs or
```
data-*
```
attributes

For each source found, describe:

The CSS selector or XPath that reaches the data element
The post-processing needed (text extraction, regex, JSON path, type conversion)
Reliability: is the selector unique and stable, or fragile?
A small HTML snippet showing the relevant element in context (use
```
...
```
to shorten long content)

Then recommend the best extraction method and explain why.

对于列表页（
is_list_page = true
）：页面包含多个重复的条目元素。你无需定位单个字段值，而是必须确定：

容器选择器 — 包裹每个条目的重复元素的CSS选择器（例如
```
article.product_pod
```
、
```
li.col-xs-6 article
```
、
```
div.item
```
）。寻找包含所有请求字段的最小重复元素。在分析输出中将其报告为
```
container_selector
```
（见步骤4）。
相对于容器的字段选择器 — 对于每个字段，记录在单个容器元素内有效的CSS选择器（例如，如果容器是
```
article.product_pod
```
，标题的相对选择器可能是
```
h3 a::attr(title)
```
）。验证该相对选择器在页面上的多个容器实例中是否一致有效。
预期条目数量 — 记录找到的容器元素数量；该数量应与values数组中的条目数量匹配。

列表页的分析应描述基于容器的提取模式，以便

scrape-codegen-generate

可以生成

for container in self.css(...):

循环。

对于详情页： 对于schema中的每个字段，生成详细分析。考虑HTML中的所有可能数据源：

HTML元素：标签、类、ID、属性 — 记录CSS选择器或XPath
JSON-LD：
```
<script type="application/ld+json">
```
块 — 记录JSON路径
Microdata：
```
itemscope
```
/
```
itemprop
```
属性
OpenGraph：
```
<meta property="og:...">
```
标签
其他脚本标签：
```
<script>
```
标签中的嵌入式JSON（例如
```
window.__DATA__ = {...}
```
）
URL组件：可从页面URL派生的数据
Meta标签：
```
<meta name="...">
```
标签
隐藏输入或
```
data-*
```
属性

对于找到的每个数据源，描述：

定位数据元素的CSS选择器或XPath
需要的后处理（文本提取、正则表达式、JSON路径、类型转换）
可靠性：选择器是否唯一且稳定，还是易失效？
显示相关元素上下文的小型HTML片段（使用
```
...
```
缩短长内容）

然后推荐最佳提取方法并解释原因。

3. Determine target values

3. 确定目标值

For each field, determine the correct target extraction value:

If expected values are provided, verify them against what's actually in the HTML
If they match, use them
If they seem wrong or incomplete, note the discrepancy and provide the corrected value based on the HTML
If a field has no data in this page, set to
```
null
```

对于每个字段，确定正确的目标提取值：

如果提供了预期值，将其与HTML中的实际内容进行验证
如果匹配，则使用该值
如果预期值看起来错误或不完整，记录差异并提供基于HTML的修正值
如果此字段在页面中无数据，设置为
```
null
```

4. Save analysis

4. 保存分析结果

Save to

{work_path}/codegen-analyze/{page_id}.json

For detail pages:

json

{
  "url": "https://example.com/product/widget-x",
  "page_id": "detail-1",
  "fields": {
    "name": {
      "target_value": "Widget X",
      "analysis": "The product name appears in two places:\n\n1. **HTML element** `<h1 class=\"product-title\">Widget X</h1>`\n   - Selector: `h1.product-title::text`\n   - Clean text, no post-processing needed\n   - Reliable: unique h1 on the page\n\n2. **JSON-LD** in `<script type=\"application/ld+json\">`:\n   ```json\n   {\"@type\": \"Product\", \"name\": \"Widget X\", ...}\n   ```\n   - Path: `name` on the Product object\n   - Also reliable\n\nRecommended: CSS selector `h1.product-title::text` — simplest, most direct."
    },
    "price": {
      "target_value": "$29.99",
      "analysis": "..."
    }
  }
}

For list pages, include

container_selector

and

item_count

, and per-field analyses use relative selectors inside the container:

json

{
  "url": "https://example.com/category/widgets/",
  "page_id": "list-1",
  "is_list_page": true,
  "container_selector": "article.product_pod",
  "item_count": 20,
  "fields": {
    "name": {
      "target_values": ["Widget X", "Widget Y", "..."],
      "analysis": "Container: article.product_pod\nRelative selector: h3 a::attr(title)\nFinds the full product title in the anchor's title attribute.\nVerified across 20 containers on the page."
    },
    "price": {
      "target_values": ["$29.99", "$14.99", "..."],
      "analysis": "Container: article.product_pod\nRelative selector: p.price_color::text\nFinds the price text directly. Verified across 20 containers."
    }
  }
}

保存至

{work_path}/codegen-analyze/{page_id}.json

。

详情页的格式：

json

{
  "url": "https://example.com/product/widget-x",
  "page_id": "detail-1",
  "fields": {
    "name": {
      "target_value": "Widget X",
      "analysis": "产品名称出现在两个位置：\n\n1. **HTML元素** `<h1 class=\"product-title\">Widget X</h1>`\n   - 选择器：`h1.product-title::text`\n   - 文本已清理，无需后处理\n   - 可靠性：页面上唯一的h1元素\n\n2. **JSON-LD** 在 `<script type=\"application/ld+json\">` 中：\n   ```json\n   {\"@type\": \"Product\", \"name\": \"Widget X\", ...}\n   ```\n   - 路径：Product对象中的`name`\n   - 同样可靠\n\n推荐：使用CSS选择器 `h1.product-title::text` — 最简单、最直接。"
    },
    "price": {
      "target_value": "$29.99",
      "analysis": "..."
    }
  }
}

列表页的格式： 需包含

container_selector

和

item_count

，且每个字段的分析使用容器内的相对选择器：

json

{
  "url": "https://example.com/category/widgets/",
  "page_id": "list-1",
  "is_list_page": true,
  "container_selector": "article.product_pod",
  "item_count": 20,
  "fields": {
    "name": {
      "target_values": ["Widget X", "Widget Y", "..."],
      "analysis": "容器：article.product_pod\n相对选择器：h3 a::attr(title)\n在锚点的title属性中找到完整产品标题。\n已在页面上的20个容器中验证。"
    },
    "price": {
      "target_values": ["$29.99", "$14.99", "..."],
      "analysis": "容器：article.product_pod\n相对选择器：p.price_color::text\n直接找到价格文本。已在20个容器中验证。"
    }
  }
}

5. Return summary

5. 返回摘要

For detail pages, return a compact summary:

detail-1 (https://...):
  name: "Widget X" — h1.product-title, also in JSON-LD
  price: "$29.99" — span.price::text, JSON-LD offers.price
  description: "A premium widget..." (2340 chars) — div.description
  rating: null — not found in HTML

For list pages, include the container selector and item count:

list-1 (https://...): 20 items, container: article.product_pod
  name: h3 a::attr(title) — "Widget X", "Widget Y", ...
  price: p.price_color::text — "$29.99", "$14.99", ...

对于详情页，返回简洁摘要：

detail-1 (https://...):
  name: "Widget X" — h1.product-title，同时存在于JSON-LD中
  price: "$29.99" — span.price::text，JSON-LD中的offers.price
  description: "A premium widget..." (2340 chars) — div.description
  rating: null — HTML中未找到

对于列表页，需包含容器选择器和条目数量：

list-1 (https://...): 20个条目，容器：article.product_pod
  name: h3 a::attr(title) — "Widget X", "Widget Y", ...
  price: p.price_color::text — "$29.99", "$14.99", ...