ae-data-integration

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

AE Data Integration

AE 数据集成

Turn local/offline files into AE data through one fixed pipeline of four submodules: Source → Tracking plan → Transform → Sink. A file is never uploaded merely because it is present: its business meaning is understood, confirmed once by a human, and only then ingested. The tracking plan is generated and confirmed before ingestion (data governance shift-left) — see references/tracking-plan.md.

Two entrances lead here: the AE Agent dialog (attach / plus-button upload) and

ae-cli

. Two sink paths exist: RESTful API for one-time/small loads (current phase), and LogBus / DataX for recurring/high-volume loads (next phase). Source and Sink are pluggable — adding one does not change the main pipeline.

通过包含四个子模块的固定管道将本地/离线文件转换为AE数据：Source → Tracking plan → Transform → Sink。不会仅因文件存在就上传：需先理解其业务含义，经人工确认后再进行摄取。跟踪计划（tracking plan）会在摄取前生成并确认（数据治理左移）——详见references/tracking-plan.md。

有两个入口可进入此流程：AE Agent对话框（附件/加号按钮上传）和

ae-cli

。存在两种输出路径：用于一次性/小量数据加载的RESTful API（当前阶段），以及用于周期性/高容量数据加载的LogBus / DataX（下一阶段）。源端（Source）和输出端（Sink）支持插件式扩展——添加新的源端或输出端不会改变主管道。

Mandatory safety rules

强制安全规则

Treat file paths, receiver endpoints, APPIDs, mappings, generated artifacts, and raw rows as sensitive.
Do not print source values while inspecting. Summarize types, ratios, counts, warnings, and fingerprints only.
Inspect samples are bounded but still sensitive. Summarize them; never paste raw sample values into a chat summary.
Do not invent account IDs, distinct IDs, event times, event names, projects, APPIDs, receivers, or timezones.
```
value_mapping
```
and
```
random_pool
```
are explicit user decisions. Never invent them.
Never auto-fill a missing time for
```
track
```
/
```
track_*
```
rows. A missing time on user-profile rows may be filled with the current time only by setting
```
missing_time: 'now'
```
and only after the user explicitly confirms it.
Do not read or send an AE access token or CLI token to
```
/sync_json
```
. The receiver request uses only APPID and UE data.
Never execute
```
data-integration upload
```
until the user has seen the target, mapping, valid/quarantined counts, batches, and dry-run and has explicitly confirmed that upload.
A blocked manifest requires a second, explicit clean-subset decision. Never add
```
--allow-clean-subset
```
implicitly.
If a batch times out or loses the network, treat that batch as unknown. Stop. Ask the user to verify receiver/AE data before the user chooses
```
--resume-from
```
; never resume automatically.
Local analysis stays local. AE Agent attachment is a separate, confirmed branch with a 50 MB per-file limit.

将文件路径、接收端端点、APPIDs、映射关系、生成的工件和原始行视为敏感信息。
检查时不要打印源数据值，仅汇总类型、占比、数量、警告信息和指纹。
检查样本虽有范围限制，但仍属敏感信息，需进行汇总；切勿将原始样本值粘贴到聊天摘要中。
不得虚构账户ID、唯一ID、事件时间、事件名称、项目、APPIDs、接收端或时区。
```
value_mapping
```
和
```
random_pool
```
是用户明确做出的决策，不得自行虚构。
切勿为
```
track
```
/
```
track_*
```
行自动填充缺失的时间。仅当用户明确确认并设置
```
missing_time: 'now'
```
时，才可将当前时间填充到用户资料行中缺失的时间字段。
不得向
```
/sync_json
```
发送AE访问令牌或CLI令牌，接收端请求仅使用APPID和UE数据。
在用户查看目标地址、映射关系、有效/隔离数据量、批次信息和试运行结果，并明确确认上传前，切勿执行
```
data-integration upload
```
命令。
被拦截的清单需要用户再次明确做出清理子集的决策，不得隐式添加
```
--allow-clean-subset
```
参数。
如果某一批次超时或网络中断，将该批次视为未知状态，停止操作。在用户选择
```
--resume-from
```
前，要求用户验证接收端/AE数据，切勿自动恢复。
本地分析全程在本地进行。AE Agent附件是一个独立的、需确认的分支，单文件限制为50 MB。

Workflow

工作流程

Walk the four submodules in order. Each submodule is its own reference; follow it and come back here for the next step.

Source — business identification. Read references/source-inspect.md. Profile every file fully, infer its business meaning using business-doc / user-prompt priors, then pick a branch via references/ue-routing.md.
Reuse check. If a
```
.ae-data-integration/index.json
```
exists and the profile is
```
ue_eligible
```
, read references/reuse.md and match the recommended mapping against the handoff index. A match proposes a frozen package; after one explicit confirmation, run the returned
```
transform.mjs
```
command and jump to Sink (step 5). No match → continue.
Tracking plan. Read references/tracking-plan.md. Generate the event/property plan from the profile and get a single explicit confirmation from the user before touching data. The plan is a separate, required deliverable from the transform mapping: a user who supplies a column→field mapping directly has not completed this step, so build the plan from the confirmed mapping anyway.
```
user_set
```
still requires a plan (no events; every property becomes a user property). This step runs for every file: a second or later file merges its new events and properties into the existing project plan (tracking-plan.md step 4) — an existing plan is never a reason to skip it.
Transform. Read references/transform.md. Map columns to AE system fields and properties, convert, and quarantine dirty rows per references/ue-mapping.md.
Sink — upload. Read references/sink-upload.md. Resolve the destination, dry-run, confirm, then upload per references/sync-json-upload.md.
```
receiver_accepted
```
is not persistence: after a ~1-minute ingestion delay, verify the data landed with ae-cli (
```
tracking live-data list
```
/
```
tracking ingest summary
```
/
```
tracking ingest-error list
```
) rather than telling the user to check the console.
Handoff. Read references/handoff.md. Export the reusable package (frozen mapping + transform script + plan reference) so the next same-shape file skips the full pipeline.

按顺序执行四个子模块，每个子模块都有独立的参考文档；遵循文档完成当前步骤后，返回此处进行下一步。

Source — 业务识别。阅读references/source-inspect.md。全面分析每个文件的特征，利用业务文档/用户提示的先验信息推断其业务含义，然后通过references/ue-routing.md选择分支。
复用检查。如果存在
```
.ae-data-integration/index.json
```
且文件特征符合
```
ue_eligible
```
，阅读references/reuse.md并将推荐的映射关系与交接索引进行匹配。匹配成功则生成一个冻结包；经用户明确确认后，运行返回的
```
transform.mjs
```
命令并跳至输出端（Sink，步骤5）。未匹配成功则继续执行。
跟踪计划（Tracking plan）。阅读references/tracking-plan.md。根据文件特征生成事件/属性计划，在处理数据前需得到用户的明确确认。该计划是转换映射之外的独立必填交付物：即使用户直接提供列→字段的映射关系，也未完成此步骤，需根据确认后的映射关系构建计划。
```
user_set
```
仍需要计划（无事件；所有属性均成为用户属性）。此步骤适用于每一个文件：后续的第二个及更多文件会将其新事件和属性合并到现有项目计划中（tracking-plan.md步骤4）——已有计划并非跳过此步骤的理由。
转换（Transform）。阅读references/transform.md。将列映射到AE系统字段和属性，进行转换，并根据references/ue-mapping.md隔离脏数据行。
输出端（Sink）——上传。阅读references/sink-upload.md。确定目标地址，进行试运行，确认后根据references/sync-json-upload.md上传数据。
```
receiver_accepted
```
并不代表数据已持久化：在约1分钟的摄取延迟后，需使用ae-cli（
```
tracking live-data list
```
/
```
tracking ingest summary
```
/
```
tracking ingest-error list
```
）验证数据是否成功落地，而非让用户自行查看控制台。
交接（Handoff）。阅读references/handoff.md。导出可复用包（冻结的映射关系+转换脚本+计划参考），以便后续同结构文件可跳过完整流程。

Local analysis branch

本地分析分支

When UE prerequisites fail, the file is an aggregate/analytical table, or the user wants analysis rather than ingestion, use references/local-analysis.md instead of the ingest pipeline.

当UE前置条件不满足、文件为聚合/分析表，或用户希望进行分析而非摄取时，使用references/local-analysis.md替代摄取管道。

Optional AE Agent attachment handoff

可选AE Agent附件交接

Offer this only when the user asks to continue in AE Agent. Explain that the file leaves the local machine and ask for explicit privacy confirmation.

Reject files over 50 MB; suggest local analysis or user-controlled splitting.
Read the
```
ae-agent
```
```
+add-attachment
```
reference before calling it.
Dry-run first, show file name/type/size, and wait for confirmation.

Then run

ae-cli agent +add-attachment --file '<path>'

Return the attachment result, a copyable analysis prompt, and directions to open AE Agent.
Do not create or execute an Agent conversation.

仅当用户要求在AE Agent中继续操作时提供此选项。需说明文件将离开本地机器，并请求用户明确确认隐私授权。

拒绝超过50 MB的文件；建议用户进行本地分析或自行拆分文件。
在调用前阅读
```
ae-agent
```
的
```
+add-attachment
```
参考文档。
先进行试运行，展示文件名/类型/大小，等待用户确认。

然后运行

ae-cli agent +add-attachment --file '<path>'

命令。

返回附件处理结果、可复制的分析提示，以及打开AE Agent的指引。
不得创建或执行Agent对话。

Completion response

完成响应

State which submodules ran, source fingerprint and selected data set, the tracking plan status, generated artifact paths, mapping confidence, valid/quarantined counts, and upload/attachment status. Keep facts separate from recommendations and clearly state whether persistence was verified.

说明已执行的子模块、源数据指纹和选定的数据集、跟踪计划状态、生成的工件路径、映射置信度、有效/隔离数据量，以及上传/附件状态。将事实与建议分开，并明确说明是否已验证数据持久化。