elasticsearch-ingest

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Elasticsearch File Ingest

Elasticsearch 文件导入

Load local data files into Elasticsearch by converting them to bulk NDJSON, creating an index with the right mappings when types matter, bulk-indexing documents, and verifying the outcome.

通过将本地数据文件转换为批量NDJSON格式、创建带有正确映射的索引（当类型十分重要时）、批量索引文档并验证结果，将本地数据文件导入Elasticsearch。

Environment Configuration

环境配置

This skill executes Elasticsearch operations through the

elastic

CLI. If the

elastic

CLI is not installed, tell the user what it is needed for. Do not guess credentials, call the HTTP API directly, or attempt other workarounds.

This skill references operations in HTTP-shorthand form (e.g.,

GET /

GET /_cat/indices

GET /{index}/_mapping

GET /{index}/_settings/index.mode

POST /_query

). The Operations table at the end of this document maps each shorthand to the equivalent

elastic

CLI command — always use the CLI rather than calling the HTTP API directly.

本技能通过

elastic

CLI执行Elasticsearch操作。如果未安装

elastic

CLI，请告知用户其用途。请勿猜测凭证、直接调用HTTP API或尝试其他变通方法。

本技能以HTTP简写形式引用操作（例如：

GET /

、

GET /_cat/indices

、

GET /{index}/_mapping

、

GET /{index}/_settings/index.mode

、

POST /_query

）。本文档末尾的操作表格将每个简写映射为对应的

elastic

CLI命令——请始终使用CLI而非直接调用HTTP API。

Scope

适用范围

This skill covers file → index loading through

POST /_bulk

. It does not use Logstash, Filebeat, Elastic Agent, Node.js ingest tools, or other sidecar pipelines. For copying documents between existing indices, use index-to-index reindex instead of re-parsing source files.

Supported source shapes:

Source shape	Example	Bulk requirement
CSV with header row	`id,name,age,...` then data rows	Parse header into field names; emit one action line + one JSON object per data row
JSON array file	`[{"a":1},{"a":2}]`	Split into per-document lines — never bulk-load the raw array as a single document
NDJSON / JSON Lines	one JSON object per line	Optionally add action lines if missing; otherwise ready for bulk

Parquet, Arrow, and other binary columnar formats are out of scope unless the user converts them to CSV or JSON first.

本技能涵盖通过

POST /_bulk

实现的文件→索引加载操作。不使用Logstash、Filebeat、Elastic Agent、Node.js导入工具或其他边车管道。如需在现有索引之间复制文档，请使用索引间重新索引，而非重新解析源文件。

支持的源文件格式：

源文件格式	示例	批量导入要求
带表头的CSV文件	`id,name,age,...` 后跟数据行	将表头解析为字段名；每一行数据对应输出一行操作指令+一行JSON对象
JSON数组文件	`[{"a":1},{"a":2}]`	拆分为每行一个文档——切勿将原始数组作为单个文档批量加载
NDJSON / JSON行格式	每行一个JSON对象	如果缺少操作行则可选添加；否则可直接用于批量导入

除非用户先将Parquet、Arrow和其他二进制列格式转换为CSV或JSON，否则不在本技能的处理范围内。

Process

操作流程

Confirm connectivity. Call
```
GET /
```
. If the call fails, stop and resolve CLI configuration before reading files or mutating cluster state.
Inspect the source file and classify its shape. Open the file (or sample the first lines) and decide:
- CSV — first line is a comma-separated header; subsequent lines are records. Count data rows (exclude the header) — you will report this count after load.
- JSON array — file starts with
```
[
```
  and contains an array of objects. Count array elements — each element becomes one indexed document, not one.
- NDJSON — one JSON value per line; lines alternate action metadata and document source, or each line is a document that still needs a preceding action line.
The decision: pick the conversion path from NDJSON Bulk Format. Never send raw CSV text or a raw JSON array body to
```
POST /_bulk
```
.
Choose the target index name. Use the name the user supplied, or propose a lowercase name derived from the file. Index names must be lowercase, cannot contain spaces or
```
/
```
, and should not start with
```
-
```
,
```
_
```
, or
```
+
```
.
Decide whether an explicit mapping is required. Call
```
GET /{index}/_mapping
```
if the index may already exist.
Create an explicit mapping before bulk loading when:
- CSV columns include numbers, dates, or booleans that must be queryable as typed fields (not plain text).
- The user asks for usable column types or aggregation-friendly fields.
- A prior load indexed everything as
```
text
```
  /
```
keyword
```
  strings and must be corrected.
When every field can remain string-like and the user did not specify types, dynamic mapping on first bulk ingest may suffice — but prefer explicit mappings for CSV unless the user explicitly accepts all-string typing.
Read Mapping Design for Ingest for type choices. When the index exists with wrong types, ask the user before calling
```
DELETE /{index}
```
and recreating it.
Create the index when needed. When step 4 requires explicit types (or the index does not exist), call
```
PUT /{index}
```
with a
```
mappings
```
block before bulk loading. Do not rely on dynamic mapping to infer
```
long
```
,
```
date
```
, or
```
boolean
```
from CSV string cells — dynamic mapping often maps ambiguous strings to
```
text
```
with a
```
.keyword
```
sub-field.
Convert the file to bulk NDJSON. Write a temporary NDJSON file where each document occupies two lines:
- Line 1 — action metadata, e.g.
```
{"index":{"_index":"<index>"}}
```
  (add
```
"_id"
```
  only when the user requires stable IDs).
- Line 2 — document JSON with correctly typed values (numbers as JSON numbers, booleans as
```
true
```
  /
```
false
```
  , dates as ISO-8601 strings such as
```
2023-01-15
```
  ).
For CSV, map the header row to JSON field names and convert cell values to the JSON types that match the mapping from step 5. For JSON arrays, iterate each array element and emit the action line + object line pair. See worked examples in NDJSON Bulk Format.
Bulk index the documents. Call
```
POST /_bulk
```
with the NDJSON file produced in step 6. Inspect the response: if
```
errors
```
is
```
true
```
, read per-item
```
error
```
objects, fix mapping or document issues, and retry failed items after remediation. Do not assume success from a zero exit code alone.
Verify the outcome. Always confirm the load — never report counts from file inspection alone.
- Call
```
GET /{index}/_count
```
  and compare to the expected row/element count from step 2.
- When typed columns matter, call
```
GET /{index}/_mapping
```
  and confirm fields such as
```
age
```
  are numeric (
```
long
```
  /
```
integer
```
  ), dates are
```
date
```
  , and booleans are
```
boolean
```
  — not
```
text
```
  .
Report the verified document count and, when relevant, the confirmed field types. If count or mapping checks fail, see Troubleshooting.

确认连接性。调用
```
GET /
```
。如果调用失败，请先解决CLI配置问题，再读取文件或修改集群状态。
检查源文件并分类其格式。打开文件（或查看前几行样本）并确定：
- CSV文件——第一行是逗号分隔的表头；后续行是记录。统计数据行数（排除表头）——加载完成后需报告此数量。
- JSON数组文件——文件以
```
[
```
  开头，包含对象数组。统计数组元素数量——每个元素对应一个索引文档，而非整体作为一个文档。
- NDJSON文件——每行一个JSON值；行交替为操作元数据和文档源，或每行是仍需前置操作行的文档。
决策：从NDJSON批量格式中选择转换路径。切勿将原始CSV文本或原始JSON数组体发送至
```
POST /_bulk
```
。
选择目标索引名称。使用用户提供的名称，或根据文件名生成小写名称。索引名称必须为小写，不能包含空格或
```
/
```
，且不应以
```
-
```
、
```
_
```
或
```
+
```
开头。
确定是否需要显式映射。如果索引可能已存在，调用
```
GET /{index}/_mapping
```
。
在以下情况下，需在批量加载之前创建显式映射：
- CSV列包含数字、日期或布尔值，且这些字段必须作为类型化字段（而非纯文本）可查询。
- 用户要求可用的列类型或适合聚合的字段。
- 之前的加载将所有字段索引为
```
text
```
  /
```
keyword
```
  字符串，需要修正。
当所有字段可保持类字符串类型且用户未指定类型时，首次批量导入时的动态映射可能足够——但除非用户明确接受全字符串类型，否则CSV文件优先使用显式映射。
阅读导入映射设计了解类型选择。当索引存在错误类型时，在调用
```
DELETE /{index}
```
并重新创建之前，请先询问用户。
必要时创建索引。当步骤4需要显式类型（或索引不存在）时，在批量加载之前调用
```
PUT /{index}
```
并附带
```
mappings
```
块。请勿依赖动态映射从CSV字符串单元格推断
```
long
```
、
```
date
```
或
```
boolean
```
类型——动态映射通常会将模糊字符串映射为带有
```
.keyword
```
子字段的
```
text
```
类型。
将文件转换为批量NDJSON格式。写入临时NDJSON文件，其中每个文档占用两行：
- 第1行——操作元数据，例如
```
{"index":{"_index":"<index>"}}
```
  （仅当用户需要稳定ID时添加
```
"_id"
```
  ）。
- 第2行——带有正确类型值的文档JSON（数字为JSON数字，布尔值为
```
true
```
  /
```
false
```
  ，日期为ISO-8601格式字符串，例如
```
2023-01-15
```
  ）。
对于CSV文件，将表头行映射为JSON字段名，并将单元格值转换为与步骤5中映射匹配的JSON类型。对于JSON数组，遍历每个数组元素并输出操作行+对象行对。请参阅NDJSON批量格式中的示例。
批量索引文档。调用
```
POST /_bulk
```
并传入步骤6生成的NDJSON文件。检查响应：如果
```
errors
```
为
```
true
```
，请读取每个条目的
```
error
```
对象，修复映射或文档问题，修复后重试失败条目。切勿仅通过退出码为0就假设操作成功。
验证结果。始终确认加载结果——切勿仅根据文件检查报告数量。
- 调用
```
GET /{index}/_count
```
  并与步骤2中的预期行/元素数量进行比较。
- 当类型化列十分重要时，调用
```
GET /{index}/_mapping
```
  并确认
```
age
```
  等字段为数值类型（
```
long
```
  /
```
integer
```
  ）、日期为
```
date
```
  类型、布尔值为
```
boolean
```
  类型——而非
```
text
```
  类型。
报告已验证的文档数量，相关时还需报告已确认的字段类型。如果数量或映射检查失败，请参阅故障排除。

Guidelines

指导原则

Bulk only. All file loads go through
```
POST /_bulk
```
with NDJSON action lines — not single-document
```
PUT
```
loops for batch files, not ingest pipelines as a substitute for client-side CSV parsing, and not posting the untouched source file.
JSON arrays must be split. A four-element array bulk-loaded as one document yields count
```
1
```
; the correct load yields count
```
4
```
.
CSV header is schema. The first CSV row names fields; each remaining row is one document. A file with one header plus five data rows must report count
```
5
```
after ingest.
Type coercion happens in the document JSON. CSV cells arrive as strings; when mappings declare
```
long
```
,
```
date
```
, or
```
boolean
```
, emit JSON numbers, ISO date strings, and boolean literals in the bulk body — do not rely on Elasticsearch to infer types from quoted CSV strings after dynamic mapping chose
```
text
```
.
Prefer explicit mappings for typed CSV. Creating the index with
```
PUT /{index}
```
first prevents silent all-text indexing that breaks range queries and aggregations.
Idempotent re-loads. When reloading into an existing index, ask the user before deleting data. Duplicate bulk
```
index
```
actions append new documents unless
```
_id
```
is specified.

仅使用批量方式。所有文件加载均通过带有NDJSON操作行的
```
POST /_bulk
```
完成——对于批量文件，不使用单文档
```
PUT
```
循环；不使用导入管道替代客户端CSV解析；不提交未修改的源文件。
JSON数组必须拆分。将包含四个元素的数组作为单个文档批量加载会得到数量
```
1
```
；正确加载会得到数量
```
4
```
。
CSV表头即为 schema。CSV的第一行定义字段名；后续每一行是一个文档。包含一个表头加五行数据的文件在导入后必须报告数量
```
5
```
。
类型转换在文档JSON中完成。CSV单元格以字符串形式存在；当映射声明为
```
long
```
、
```
date
```
或
```
boolean
```
时，在批量体中输出JSON数字、ISO日期字符串和布尔字面量——切勿依赖Elasticsearch在动态映射选择
```
text
```
类型后，从带引号的CSV字符串中推断类型。
CSV文件优先使用显式映射。先通过
```
PUT /{index}
```
创建索引可避免静默全文本索引，这种情况会破坏范围查询和聚合操作。
幂等重新加载。当重新加载到现有索引时，请在删除数据前询问用户。除非指定
```
_id
```
，否则重复的批量
```
index
```
操作会追加新文档。

Examples

示例

CSV with typed columns

带类型化列的CSV文件

Source (

users.csv

— header + 5 data rows):

csv

id,name,age,signup_date,active
1,Ada Lovelace,36,2023-01-15,true

Create the index with explicit types, convert rows to NDJSON (five action+document pairs for five data rows), bulk load, then verify count

and mapping types. Full walkthrough: Mapping Design for Ingest and NDJSON Bulk Format.

源文件（

users.csv

——表头+5行数据）：

csv

id,name,age,signup_date,active
1,Ada Lovelace,36,2023-01-15,true

创建带有显式类型的索引，将行转换为NDJSON格式（五行数据对应五组操作+文档对），批量加载，然后验证数量为

并检查映射类型。完整教程：导入映射设计和NDJSON批量格式。

JSON array file

JSON数组文件

Source (

events.json

json

[
  { "event_id": "e-1", "type": "login", "user_id": 1, "value": 12.5 },
  { "event_id": "e-2", "type": "logout", "user_id": 1, "value": 0.0 }
]

Convert to four bulk line pairs for four array elements (not one pair for the whole array). Verify

GET /{index}/_count

returns

. See NDJSON Bulk Format.

源文件（

events.json

）：

json

[
  { "event_id": "e-1", "type": "login", "user_id": 1, "value": 12.5 },
  { "event_id": "e-2", "type": "logout", "user_id": 1, "value": 0.0 }
]

转换为四组批量行对（对应四个数组元素，而非整个数组一组行对）。验证

GET /{index}/_count

。请参阅NDJSON批量格式。

NDJSON already prepared

已准备好的NDJSON文件

When the file alternates action lines and document lines, validate the format and pass it directly to

POST /_bulk

after confirming the target index and mappings.

当文件交替为操作行和文档行时，验证格式后，在确认目标索引和映射后直接传入

POST /_bulk

。

When Not to Use

不适用场景

Continuous or streaming ingestion — use Elastic Agent or Beats to tail logs and metrics.
Complex enrichment pipelines — design server-side ingest pipelines separately; this skill still converts files to bulk NDJSON client-side before load.
Index-to-index copy or mapping migration — reindex between indices instead of exporting to files.
Very large binary columnar files — convert to CSV or JSON offline first, then follow this skill.

持续或流式导入——使用Elastic Agent或Beats来追踪日志和指标。
复杂 enrichment 管道——单独设计服务器端导入管道；本技能仍会在客户端将文件转换为批量NDJSON格式后再加载。
索引间复制或映射迁移——在索引间重新索引，而非导出到文件。
超大二进制列文件——先离线转换为CSV或JSON，再遵循本技能操作。

References

参考资料

NDJSON Bulk Format — CSV and JSON-array conversion, action-line syntax, batch sizing
Mapping Design for Ingest — explicit mappings for CSV types, eval-style schemas
Troubleshooting — wrong counts, text-typed numerics, bulk item errors

NDJSON批量格式——CSV和JSON数组转换、操作行语法、批量大小
导入映射设计——CSV类型的显式映射、评估式schema
故障排除——数量错误、文本类型数值、批量条目错误

Operations

操作

HTTP API (shorthand)	`elastic` CLI command
`GET /`	`elastic es info`
`PUT /{index}`	`elastic es indices create --index '<index>' --mappings '<json>'`
`DELETE /{index}`	`elastic es indices delete --index '<index>'`
`POST /_bulk`	`elastic es bulk --index '<index>' --input-file '<ndjson-path>'`
`GET /{index}/_count`	`elastic es count --index '<index>'`
`GET /{index}/_mapping`	`elastic es indices get-mapping --index '<index>'`

HTTP API（简写）	`elastic` CLI 命令
`GET /`	`elastic es info`
`PUT /{index}`	`elastic es indices create --index '<index>' --mappings '<json>'`
`DELETE /{index}`	`elastic es indices delete --index '<index>'`
`POST /_bulk`	`elastic es bulk --index '<index>' --input-file '<ndjson-path>'`
`GET /{index}/_count`	`elastic es count --index '<index>'`
`GET /{index}/_mapping`	`elastic es indices get-mapping --index '<index>'`