Back to Details

oxylabs-web-scraper

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

Oxylabs Web Scraper API

Oxylabs Web Scraper API

Authentication

身份验证

Requires HTTP Basic Auth with credentials from environment variables:
bash
curl -u "$OXY_WSA_USERNAME:$OXY_WSA_PASSWORD" ...
需要使用环境变量中的凭证进行HTTP Basic Auth认证：
bash
curl -u "$OXY_WSA_USERNAME:$OXY_WSA_PASSWORD" ...

Endpoint

端点

POST https://realtime.oxylabs.io/v1/queries   # immediate response
POST https://data.oxylabs.io/v1/queries       # Push-Pull jobs, callbacks, storage
Content-Type: application/json
POST https://realtime.oxylabs.io/v1/queries   # 即时响应
POST https://data.oxylabs.io/v1/queries       # 推拉式任务、回调、存储
Content-Type: application/json

Core Parameters

核心参数

ParameterRequiredDescription
source
YesTarget scraper (e.g., 
universal
, 
amazon_product
, 
google_search
)
url
ConditionalURL to scrape (for 
universal
and 
*_url
sources)
query
ConditionalSearch query or product ID (for 
*_search
and 
*_product
sources)
parse
NoEnable structured data parsing (recommended for supported sources)
render
NoJavaScript rendering: 
html
or 
png
geo_location
NoGeographic targeting: country/state/city, ZIP/postcode, coordinates, or Criteria ID where supported
session_id
NoReuse the same proxy IP across multiple jobs
content_encoding
NoSet to 
base64
when downloading image files via Realtime or Push-Pull
user_agent_type
NoDevice/browser preset, e.g., 
desktop_chrome
, 
mobile_ios
, 
tablet_android
locale
NoInterface language / 
Accept-Language
, e.g., 
de-DE
callback_url
NoPush-Pull callback endpoint
storage_type
, 
storage_url
NoPush-Pull cloud upload target (
gcs
, 
s3
, 
tos
, 
s3_compatible
)
markdown
, 
xhr
NoEnable markdown or captured XHR result types
browser_instructions
NoRendered browser actions; requires 
render: "html"
parsing_instructions
, 
parser_preset
NoCustom parser rules or saved preset; pair with 
parse: true
client_notes
NoClient-side job tag saved with the job metadata
domain
, 
subdomain
, 
start_page
, 
pages
, 
limit
, 
store_id
, 
delivery_zip
, 
fulfillment_type
Source-specificMarketplace/search/store localization and pagination fields
user_agent_type
values: 
desktop
, 
desktop_chrome
, 
desktop_edge
, 
desktop_firefox
, 
desktop_opera
, 
desktop_safari
, 
mobile
, 
mobile_android
, 
mobile_ios
, 
tablet
, 
tablet_android
, 
tablet_ios
.
参数是否必填描述
source
是目标抓取器（例如：
universal
、
amazon_product
、
google_search
）
url
可选要抓取的URL（适用于
universal
和
*_url
类型的source）
query
可选搜索查询或产品ID（适用于
*_search
和
*_product
类型的source）
parse
否启用结构化数据解析（推荐在支持的source中使用）
render
否JavaScript渲染：
html
或
png
geo_location
否地理定位：国家/州/城市、邮政编码、坐标，或支持的Criteria ID
session_id
否在多个任务中复用相同的代理IP
content_encoding
否当通过即时接口或推拉式接口下载图片文件时，设置为
base64
user_agent_type
否设备/浏览器预设，例如：
desktop_chrome
、
mobile_ios
、
tablet_android
locale
否界面语言 / 
Accept-Language
，例如：
de-DE
callback_url
否推拉式任务的回调端点
storage_type
, 
storage_url
否推拉式任务的云上传目标（
gcs
、
s3
、
tos
、
s3_compatible
）
markdown
, 
xhr
否启用markdown或捕获的XHR结果类型
browser_instructions
否渲染浏览器操作；需要配合
render: "html"
使用
parsing_instructions
, 
parser_preset
否自定义解析规则或已保存的预设；需配合
parse: true
使用
client_notes
否与任务元数据一起保存的客户端任务标签
domain
, 
subdomain
, 
start_page
, 
pages
, 
limit
, 
store_id
, 
delivery_zip
, 
fulfillment_type
源特定商城/搜索/店铺本地化及分页字段
user_agent_type
可选值：
desktop
、
desktop_chrome
、
desktop_edge
、
desktop_firefox
、
desktop_opera
、
desktop_safari
、
mobile
、
mobile_android
、
mobile_ios
、
tablet
、
tablet_android
、
tablet_ios
。

Context Parameters

上下文参数

Add these as 
{ "key": "...", "value": ... }
objects in 
context
:
KeyUse
force_headers
, 
headers
Merge custom headers with managed headers
force_cookies
, 
cookies
Merge custom cookies with managed cookies
http_method
, 
content
Use 
post
with Base64-encoded body content
follow_redirects
Follow 3xx redirect chains
successful_status_codes
Treat specific non-standard HTTP codes as successful
For multi-format output, enable types in the payload (
parse
, 
markdown
, 
xhr
, 
render: "png"
) and request them with 
?type=raw,parsed,png,markdown,xhr
.
For batch Push-Pull jobs, use 
POST /v1/queries/batch
with arrays only for 
query
or 
url
; keep all other parameters singular. Maximum batch size is 5,000 values.
以
{ "key": "...", "value": ... }
对象形式添加到
context
中：
键用途
force_headers
, 
headers
将自定义请求头与管理请求头合并
force_cookies
, 
cookies
将自定义Cookie与管理Cookie合并
http_method
, 
content
使用
post
方法并传入Base64编码的请求体内容
follow_redirects
跟随3xx重定向链
successful_status_codes
将特定非标准HTTP状态码视为成功
如需多格式输出，在请求体中启用对应类型（
parse
、
markdown
、
xhr
、
render: "png"
），并通过
?type=raw,parsed,png,markdown,xhr
参数指定请求的类型。
对于批量推拉式任务，使用
POST /v1/queries/batch
接口，仅
query
或
url
参数可使用数组形式；其余所有参数需保持单个值。最大批量大小为5000个值。

Quick Start

快速开始

Scrape any URL:
bash
curl -X POST 'https://realtime.oxylabs.io/v1/queries' \
  -u "$OXY_WSA_USERNAME:$OXY_WSA_PASSWORD" \
  -H 'Content-Type: application/json' \
  -d '{"source": "universal", "url": "https://example.com"}'
Google search with parsing:
bash
curl -X POST 'https://realtime.oxylabs.io/v1/queries' \
  -u "$OXY_WSA_USERNAME:$OXY_WSA_PASSWORD" \
  -H 'Content-Type: application/json' \
  -d '{"source": "google_search", "query": "best laptops", "parse": true}'
Amazon product by ASIN:
bash
curl -X POST 'https://realtime.oxylabs.io/v1/queries' \
  -u "$OXY_WSA_USERNAME:$OXY_WSA_PASSWORD" \
  -H 'Content-Type: application/json' \
  -d '{"source": "amazon_product", "query": "B07FZ8S74R", "parse": true}'
抓取任意URL：
bash
curl -X POST 'https://realtime.oxylabs.io/v1/queries' \
  -u "$OXY_WSA_USERNAME:$OXY_WSA_PASSWORD" \
  -H 'Content-Type: application/json' \
  -d '{"source": "universal", "url": "https://example.com"}'
带解析的Google搜索：
bash
curl -X POST 'https://realtime.oxylabs.io/v1/queries' \
  -u "$OXY_WSA_USERNAME:$OXY_WSA_PASSWORD" \
  -H 'Content-Type: application/json' \
  -d '{"source": "google_search", "query": "best laptops", "parse": true}'
通过ASIN抓取Amazon产品：
bash
curl -X POST 'https://realtime.oxylabs.io/v1/queries' \
  -u "$OXY_WSA_USERNAME:$OXY_WSA_PASSWORD" \
  -H 'Content-Type: application/json' \
  -d '{"source": "amazon_product", "query": "B07FZ8S74R", "parse": true}'

Choosing the Right Source

选择合适的Source

  1. Use specific sources when available (
    amazon_product
    , 
    google_search
    ) - better parsing and reliability
  2. Use 
    universal
    for unsupported sites     - works with any URL
  3. Enable 
    parse: true
     for structured JSON output on supported sources
  1. 优先使用特定source（如
    amazon_product
    、
    google_search
    ）——解析效果更好，可靠性更高
  2. 对于不支持的站点，使用
    universal
    ——适用于任意URL
  3. 在支持的source中启用
    parse: true
    ——获取结构化JSON输出

Response Structure

响应结构

json
{
  "results": [{
    "content": "...",
    "status_code": 200,
    "url": "https://..."
  }]
}
With 
parse: true
, 
content
contains structured data (title, price, reviews, etc.) instead of raw HTML.
json
{
  "results": [{
    "content": "...",
    "status_code": 200,
    "url": "https://..."
  }]
}
当启用
parse: true
时，
content
字段将包含结构化数据（标题、价格、评论等），而非原始HTML。

Available Sources

可用Source

For the complete list of 40+ supported sources organized by category, see sources.md.
有关按类别划分的40+支持source完整列表，请查看sources.md。

More Examples

更多示例

For detailed request/response examples including geo-location, JavaScript rendering, and custom headers, see examples.md.
有关包含地理定位、JavaScript渲染和自定义请求头的详细请求/响应示例，请查看examples.md。

Error Handling

错误处理

CodeMeaning
200Success
400Invalid parameters
401Authentication failed
403Access denied
429Rate limit exceeded
状态码含义
200成功
400参数无效
401认证失败
403访问被拒绝
429超出速率限制

Key Guidelines

核心指南

  • Always set 
    parse: true
    for supported sources to get structured data
  • Use ZIP codes for US e-commerce geo-location (e.g., 
    "90210"
    )
  • Use country/state format for search engines (e.g., 
    "California,United States"
    )
  • Add 
    render: "html"
    for JavaScript-heavy pages
  • Use 
    render: ""
    only to disable automatic forced rendering for force-rendered pages; set client timeouts near 180 seconds for rendered Realtime or Proxy Endpoint requests
  • Add 
    content_encoding: "base64"
    when scraping image URLs, then decode 
    results[0].content
    before saving the file
  • 对于支持的source，始终设置
    parse: true
    以获取结构化数据
  • 针对美国电商地理定位，使用邮政编码（例如：
    "90210"
    ）
  • 针对搜索引擎，使用国家/州格式（例如：
    "California,United States"
    ）
  • 对于JavaScript密集型页面，添加
    render: "html"
  • 仅在强制渲染页面中需要禁用自动强制渲染时，设置
    render: ""
    ；对于渲染型即时请求或代理端点请求，将客户端超时设置为180秒左右
  • 抓取图片URL时，添加
    content_encoding: "base64"
    ，保存文件前需解码
    results[0].content