Loading...
Loading...
Compare original and translation side by side
XCRAWL_API_KEY~/.xcrawl/config.json{
"XCRAWL_API_KEY": "<your_api_key>"
}XCRAWL_API_KEY~/.xcrawl/config.json{
"XCRAWL_API_KEY": "<your_api_key>"
}https://dash.xcrawl.com/1000https://dash.xcrawl.com/curlnodecurlnodePOST /v1/crawlGET /v1/crawl/{crawl_id}https://run.xcrawl.comAuthorization: Bearer <XCRAWL_API_KEY>POST /v1/crawlGET /v1/crawl/{crawl_id}https://run.xcrawl.comAuthorization: Bearer <XCRAWL_API_KEY>API_KEY="$(node -e "const fs=require('fs');const p=process.env.HOME+'/.xcrawl/config.json';const k=JSON.parse(fs.readFileSync(p,'utf8')).XCRAWL_API_KEY||'';process.stdout.write(k)")"
CREATE_RESP="$(curl -sS -X POST "https://run.xcrawl.com/v1/crawl" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${API_KEY}" \
-d '{"url":"https://example.com","crawler":{"limit":100,"max_depth":2},"output":{"formats":["markdown","links"]}}')"
echo "$CREATE_RESP"
CRAWL_ID="$(node -e 'const s=process.argv[1];const j=JSON.parse(s);process.stdout.write(j.crawl_id||"")' "$CREATE_RESP")"
curl -sS -X GET "https://run.xcrawl.com/v1/crawl/${CRAWL_ID}" \
-H "Authorization: Bearer ${API_KEY}"API_KEY="$(node -e "const fs=require('fs');const p=process.env.HOME+'/.xcrawl/config.json';const k=JSON.parse(fs.readFileSync(p,'utf8')).XCRAWL_API_KEY||'';process.stdout.write(k)")"
CREATE_RESP="$(curl -sS -X POST "https://run.xcrawl.com/v1/crawl" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${API_KEY}" \
-d '{"url":"https://example.com","crawler":{"limit":100,"max_depth":2},"output":{"formats":["markdown","links"]}}')"
echo "$CREATE_RESP"
CRAWL_ID="$(node -e 'const s=process.argv[1];const j=JSON.parse(s);process.stdout.write(j.crawl_id||"")' "$CREATE_RESP")"
curl -sS -X GET "https://run.xcrawl.com/v1/crawl/${CRAWL_ID}" \
-H "Authorization: Bearer ${API_KEY}"node -e '
const fs=require("fs");
const apiKey=JSON.parse(fs.readFileSync(process.env.HOME+"/.xcrawl/config.json","utf8")).XCRAWL_API_KEY;
const body={url:"https://example.com",crawler:{limit:300,max_depth:3,include:["/docs/.*"],exclude:["/blog/.*"]},request:{locale:"ja-JP"},output:{formats:["markdown","links","json"]}};
fetch("https://run.xcrawl.com/v1/crawl",{
method:"POST",
headers:{"Content-Type":"application/json",Authorization:`Bearer ${apiKey}`},
body:JSON.stringify(body)
}).then(async r=>{console.log(await r.text());});
'node -e '
const fs=require("fs");
const apiKey=JSON.parse(fs.readFileSync(process.env.HOME+"/.xcrawl/config.json","utf8")).XCRAWL_API_KEY;
const body={url:"https://example.com",crawler:{limit:300,max_depth:3,include:["/docs/.*"],exclude:["/blog/.*"]},request:{locale:"ja-JP"},output:{formats:["markdown","links","json"]}};
fetch("https://run.xcrawl.com/v1/crawl",{
method:"POST",
headers:{"Content-Type":"application/json",Authorization:`Bearer ${apiKey}`},
body:JSON.stringify(body)
}).then(async r=>{console.log(await r.text());});
'POST https://run.xcrawl.com/v1/crawlContent-Type: application/jsonAuthorization: Bearer <api_key>POST https://run.xcrawl.com/v1/crawlContent-Type: application/jsonAuthorization: Bearer <api_key>| Field | Type | Required | Default | Description |
|---|---|---|---|---|
| string | Yes | - | Site entry URL |
| object | No | - | Crawler config |
| object | No | - | Proxy config |
| object | No | - | Request config |
| object | No | - | JS rendering config |
| object | No | - | Output config |
| object | No | - | Async callback config |
| 字段 | 类型 | 必填 | 默认值 | 描述 |
|---|---|---|---|---|
| string | 是 | - | 网站入口URL |
| object | 否 | - | 爬虫配置 |
| object | 否 | - | 代理配置 |
| object | 否 | - | 请求配置 |
| object | 否 | - | JS渲染配置 |
| object | 否 | - | 输出配置 |
| object | 否 | - | 异步回调配置 |
crawlercrawler| Field | Type | Required | Default | Description |
|---|---|---|---|---|
| integer | No | | Max pages |
| string[] | No | - | Include only matching URLs (regex supported) |
| string[] | No | - | Exclude matching URLs (regex supported) |
| integer | No | | Max depth from entry URL |
| boolean | No | | Crawl full site instead of only subpaths |
| boolean | No | | Include subdomains |
| boolean | No | | Include external links |
| boolean | No | | Use site sitemap |
| 字段 | 类型 | 必填 | 默认值 | 描述 |
|---|---|---|---|---|
| integer | 否 | | 最大爬取页面数 |
| string[] | 否 | - | 仅包含匹配的URL(支持正则) |
| string[] | 否 | - | 排除匹配的URL(支持正则) |
| integer | 否 | | 相对于入口URL的最大爬取深度 |
| boolean | 否 | | 爬取整个站点而非仅子路径 |
| boolean | 否 | | 包含子域名 |
| boolean | 否 | | 包含外部链接 |
| boolean | 否 | | 使用站点地图 |
proxyproxy| Field | Type | Required | Default | Description |
|---|---|---|---|---|
| string | No | | ISO-3166-1 alpha-2 country code, e.g. |
| string | No | Auto-generated | Sticky session ID; same ID attempts to reuse exit |
| 字段 | 类型 | 必填 | 默认值 | 描述 |
|---|---|---|---|---|
| string | 否 | | ISO-3166-1 alpha-2国家代码,例如 |
| string | 否 | 自动生成 | 粘性会话ID;相同ID会尝试复用出口节点 |
requestrequest| Field | Type | Required | Default | Description |
|---|---|---|---|---|
| string | No | | Affects |
| string | No | | |
| object map | No | - | Cookie key/value pairs |
| object map | No | - | Header key/value pairs |
| boolean | No | | Return main content only |
| boolean | No | | Attempt to block ad resources |
| boolean | No | | Skip TLS verification |
| 字段 | 类型 | 必填 | 默认值 | 描述 |
|---|---|---|---|---|
| string | 否 | | 影响 |
| string | 否 | | |
| 对象映射 | 否 | - | Cookie键值对 |
| 对象映射 | 否 | - | 请求头键值对 |
| boolean | 否 | | 仅返回页面主内容 |
| boolean | 否 | | 尝试阻止广告资源加载 |
| boolean | 否 | | 跳过TLS验证 |
js_renderjs_render| Field | Type | Required | Default | Description |
|---|---|---|---|---|
| boolean | No | | Enable browser rendering |
| string | No | | |
| integer | No | - | Viewport width (desktop |
| integer | No | - | Viewport height (desktop |
| 字段 | 类型 | 必填 | 默认值 | 描述 |
|---|---|---|---|---|
| boolean | 否 | | 启用浏览器渲染 |
| string | 否 | | |
| integer | 否 | - | 视口宽度(桌面端为 |
| integer | 否 | - | 视口高度(桌面端为 |
outputoutput| Field | Type | Required | Default | Description |
|---|---|---|---|---|
| string[] | No | | Output formats |
| string | No | | |
| string | No | - | Extraction prompt |
| object | No | - | JSON Schema |
output.formatshtmlraw_htmlmarkdownlinkssummaryscreenshotjson| 字段 | 类型 | 必填 | 默认值 | 描述 |
|---|---|---|---|---|
| string[] | 否 | | 输出格式 |
| string | 否 | | |
| string | 否 | - | 提取提示词 |
| object | 否 | - | JSON Schema |
output.formatshtmlraw_htmlmarkdownlinkssummaryscreenshotjsonwebhookwebhook| Field | Type | Required | Default | Description |
|---|---|---|---|---|
| string | No | - | Callback URL |
| object map | No | - | Custom callback headers |
| string[] | No | | Events: |
| 字段 | 类型 | 必填 | 默认值 | 描述 |
|---|---|---|---|---|
| string | 否 | - | 回调URL |
| 对象映射 | 否 | - | 自定义回调请求头 |
| string[] | 否 | | 触发事件: |
POST /v1/crawlPOST /v1/crawl| Field | Type | Description |
|---|---|---|
| string | Task ID |
| string | Always |
| string | Version |
| string | Always |
| 字段 | 类型 | 描述 |
|---|---|---|
| string | 任务ID |
| string | 固定为 |
| string | 版本号 |
| string | 固定为 |
GET /v1/crawl/{crawl_id}GET /v1/crawl/{crawl_id}| Field | Type | Description |
|---|---|---|
| string | Task ID |
| string | Always |
| string | Version |
| string | |
| string | Entry URL |
| object[] | Per-page result array |
| string | Start time (ISO 8601) |
| string | End time (ISO 8601) |
| integer | Total credits used |
data[]output.formatshtmlraw_htmlmarkdownlinkssummaryscreenshotjsonmetadatatraffic_bytescredits_usedcredits_detail| 字段 | 类型 | 描述 |
|---|---|---|
| string | 任务ID |
| string | 固定为 |
| string | 版本号 |
| string | |
| string | 入口URL |
| object[] | 单页面结果数组 |
| string | 任务开始时间(ISO 8601格式) |
| string | 任务结束时间(ISO 8601格式) |
| integer | 总消耗积分 |
data[]output.formatshtmlraw_htmlmarkdownlinkssummaryscreenshotjsonmetadatatraffic_bytescredits_usedcredits_detailcrawl_idGET /v1/crawl/{crawl_id}pendingcrawlingcompletedfailedcrawl_idGET /v1/crawl/{crawl_id}pendingcrawlingcompletedfailedPOST /v1/crawlGET /v1/crawl/{crawl_id}request_payloadPOST /v1/crawlGET /v1/crawl/{crawl_id}request_payload