huggingface-datasets

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Hugging Face Dataset Viewer

Use this skill to execute read-only Dataset Viewer API calls for dataset exploration and extraction.

本技能用于执行只读的Dataset Viewer API调用，以进行数据集探索与提取。

Core workflow

核心工作流

Optionally validate dataset availability with
```
/is-valid
```
.
Resolve
```
config
```
+
```
split
```
with
```
/splits
```
.
Preview with
```
/first-rows
```
.
Paginate content with
```
/rows
```
using
```
offset
```
and
```
length
```
(max 100).
Use
```
/search
```
for text matching and
```
/filter
```
for row predicates.
Retrieve parquet links via
```
/parquet
```
and totals/metadata via
```
/size
```
and
```
/statistics
```
.

（可选）通过
```
/is-valid
```
验证数据集是否可用。
通过
```
/splits
```
解析
```
config
```
+
```
split
```
。
通过
```
/first-rows
```
预览数据。
使用
```
offset
```
和
```
length
```
（最大值为100），通过
```
/rows
```
实现内容分页。
使用
```
/search
```
进行文本匹配，使用
```
/filter
```
进行行数据谓词筛选。
通过
```
/parquet
```
获取Parquet文件链接，通过
```
/size
```
和
```
/statistics
```
获取数据总量及元数据。

Defaults

默认配置

Base URL:
```
https://datasets-server.huggingface.co
```
Default API method:
```
GET
```
Query params should be URL-encoded.
```
offset
```
is 0-based.
```
length
```
max is usually
```
100
```
for row-like endpoints.
Gated/private datasets require
```
Authorization: Bearer <HF_TOKEN>
```
.

基础URL：
```
https://datasets-server.huggingface.co
```
默认API请求方法：
```
GET
```
查询参数需进行URL编码。
```
offset
```
从0开始计数。
类行数据接口的
```
length
```
最大值通常为
```
100
```
。
受限制/私有数据集需要携带
```
Authorization: Bearer <HF_TOKEN>
```
请求头。

Dataset Viewer

Dataset Viewer 接口说明

Validate dataset

/is-valid?dataset=<namespace/repo>

List subsets and splits

/splits?dataset=<namespace/repo>

Preview first rows

/first-rows?dataset=<namespace/repo>&config=<config>&split=<split>

Paginate rows

/rows?dataset=<namespace/repo>&config=<config>&split=<split>&offset=<int>&length=<int>

Search text

/search?dataset=<namespace/repo>&config=<config>&split=<split>&query=<text>&offset=<int>&length=<int>

Filter with predicates

/filter?dataset=<namespace/repo>&config=<config>&split=<split>&where=<predicate>&orderby=<sort>&offset=<int>&length=<int>

List parquet shards

/parquet?dataset=<namespace/repo>

Get size totals

/size?dataset=<namespace/repo>

Get column statistics

/statistics?dataset=<namespace/repo>&config=<config>&split=<split>

Get Croissant metadata (if available)

/croissant?dataset=<namespace/repo>

Pagination pattern:

bash

curl "https://datasets-server.huggingface.co/rows?dataset=stanfordnlp/imdb&config=plain_text&split=train&offset=0&length=100"
curl "https://datasets-server.huggingface.co/rows?dataset=stanfordnlp/imdb&config=plain_text&split=train&offset=100&length=100"

When pagination is partial, use response fields such as

num_rows_total

num_rows_per_page

, and

partial

to drive continuation logic.

Search/filter notes:

```
/search
```
matches string columns (full-text style behavior is internal to the API).
```
/filter
```
requires predicate syntax in
```
where
```
and optional sort in
```
orderby
```
.
Keep filtering and searches read-only and side-effect free.

验证数据集可用性

：

/is-valid?dataset=<namespace/repo>

列出子集与拆分

：

/splits?dataset=<namespace/repo>

预览前几行数据

：

/first-rows?dataset=<namespace/repo>&config=<config>&split=<split>

分页查看行数据

：

/rows?dataset=<namespace/repo>&config=<config>&split=<split>&offset=<int>&length=<int>

文本搜索

：

/search?dataset=<namespace/repo>&config=<config>&split=<split>&query=<text>&offset=<int>&length=<int>

谓词筛选

：

/filter?dataset=<namespace/repo>&config=<config>&split=<split>&where=<predicate>&orderby=<sort>&offset=<int>&length=<int>

列出Parquet分片

：

/parquet?dataset=<namespace/repo>

获取数据总量

：

/size?dataset=<namespace/repo>

获取列统计信息

：

/statistics?dataset=<namespace/repo>&config=<config>&split=<split>

获取Croissant元数据（若可用）

：

/croissant?dataset=<namespace/repo>

分页示例：

bash

curl "https://datasets-server.huggingface.co/rows?dataset=stanfordnlp/imdb&config=plain_text&split=train&offset=0&length=100"
curl "https://datasets-server.huggingface.co/rows?dataset=stanfordnlp/imdb&config=plain_text&split=train&offset=100&length=100"

当分页结果不完整时，可使用响应字段如

num_rows_total

、

num_rows_per_page

和

partial

来驱动后续的分页逻辑。

搜索/筛选注意事项：

```
/search
```
用于匹配字符串类型列（全文检索行为由API内部实现）。
```
/filter
```
需要在
```
where
```
参数中指定谓词语法，还可通过
```
orderby
```
参数设置排序规则。
确保筛选与搜索操作均为只读，无副作用。

Querying Datasets

数据集查询

Use

npx parquetlens

with Hub parquet alias paths for SQL querying.

Parquet alias shape:

text

hf://datasets/<namespace>/<repo>@~parquet/<config>/<split>/<shard>.parquet

Derive

<config>

<split>

, and

<shard>

from Dataset Viewer

/parquet

bash

curl -s "https://datasets-server.huggingface.co/parquet?dataset=cfahlgren1/hub-stats" \
  | jq -r '.parquet_files[] | "hf://datasets/\(.dataset)@~parquet/\(.config)/\(.split)/\(.filename)"'

Run SQL query:

bash

npx -y -p parquetlens -p @parquetlens/sql parquetlens \
  "hf://datasets/<namespace>/<repo>@~parquet/<config>/<split>/<shard>.parquet" \
  --sql "SELECT * FROM data LIMIT 20"

可结合Hub的Parquet别名路径，使用

npx parquetlens

进行SQL查询。

Parquet别名格式：

text

hf://datasets/<namespace>/<repo>@~parquet/<config>/<split>/<shard>.parquet

从Dataset Viewer的

/parquet

接口获取

<config>

、

<split>

和

<shard>

的值：

bash

curl -s "https://datasets-server.huggingface.co/parquet?dataset=cfahlgren1/hub-stats" \
  | jq -r '.parquet_files[] | "hf://datasets/\(.dataset)@~parquet/\(.config)/\(.split)/\(.filename)"'

执行SQL查询：

bash

npx -y -p parquetlens -p @parquetlens/sql parquetlens \
  "hf://datasets/<namespace>/<repo>@~parquet/<config>/<split>/<shard>.parquet" \
  --sql "SELECT * FROM data LIMIT 20"

SQL export

SQL导出

CSV:

--sql "COPY (SELECT * FROM data LIMIT 1000) TO 'export.csv' (FORMAT CSV, HEADER, DELIMITER ',')"

JSON:

--sql "COPY (SELECT * FROM data LIMIT 1000) TO 'export.json' (FORMAT JSON)"

Parquet:

--sql "COPY (SELECT * FROM data LIMIT 1000) TO 'export.parquet' (FORMAT PARQUET)"

CSV格式：

--sql "COPY (SELECT * FROM data LIMIT 1000) TO 'export.csv' (FORMAT CSV, HEADER, DELIMITER ',')"

JSON格式：

--sql "COPY (SELECT * FROM data LIMIT 1000) TO 'export.json' (FORMAT JSON)"

Parquet格式：

--sql "COPY (SELECT * FROM data LIMIT 1000) TO 'export.parquet' (FORMAT PARQUET)"

Creating and Uploading Datasets

数据集创建与上传

Use one of these flows depending on dependency constraints.

Zero local dependencies (Hub UI):

Create dataset repo in browser:
```
https://huggingface.co/new-dataset
```
Upload parquet files in the repo "Files and versions" page.
Verify shards appear in Dataset Viewer:

bash

curl -s "https://datasets-server.huggingface.co/parquet?dataset=<namespace>/<repo>"

Low dependency CLI flow (

npx @huggingface/hub

hfjs

Set auth token:

bash

export HF_TOKEN=<your_hf_token>

Upload parquet folder to a dataset repo (auto-creates repo if missing):

bash

npx -y @huggingface/hub upload datasets/<namespace>/<repo> ./local/parquet-folder data

Upload as private repo on creation:

bash

npx -y @huggingface/hub upload datasets/<namespace>/<repo> ./local/parquet-folder data --private

After upload, call

/parquet

to discover

<config>/<split>/<shard>

values for querying with

@~parquet

可根据依赖约束选择以下任一流程。

零本地依赖（Hub UI）：

在浏览器中创建数据集仓库：
```
https://huggingface.co/new-dataset
```
在仓库的“Files and versions”页面上传Parquet文件。
验证分片是否在Dataset Viewer中显示：

bash

curl -s "https://datasets-server.huggingface.co/parquet?dataset=<namespace>/<repo>"

低依赖CLI流程（

npx @huggingface/hub

hfjs

）：

设置认证令牌：

bash

export HF_TOKEN=<your_hf_token>

将Parquet文件夹上传至数据集仓库（若仓库不存在则自动创建）：

bash

npx -y @huggingface/hub upload datasets/<namespace>/<repo> ./local/parquet-folder data

创建私有仓库并上传：

bash

npx -y @huggingface/hub upload datasets/<namespace>/<repo> ./local/parquet-folder data --private

上传完成后，调用

/parquet

接口获取

<config>/<split>/<shard>

的值，以便使用

@~parquet

进行查询。