configuring-lookup-caches

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese
<!-- TIER:1 -->
<!-- TIER:1 -->

Configuring Lookup Caches

配置查找缓存

A lookup cache is an in-memory key-value store managed by Celigo for fast data retrieval during integration processing. Unlike export-based lookups that query external systems per-record, lookup caches provide sub-millisecond access to pre-loaded reference data without consuming API calls.
Lookup caches handle three concerns:
  • Cache metadata -- name, integration scope, and whether data is included in templates/cloning
  • Data management -- loading, retrieving, updating, and purging key-value entries via dedicated data endpoints
  • Size governance -- each cache has a 50 MB limit; the per-environment aggregate limit across all caches is 1 GB
Lookup caches are consumed by import and export mappings via the
lookups[]
array. A lookup entry references a cache by
_lookupCacheId
and optionally extracts a specific field from the cached object via the
extract
JSON path. This is configured on the import/export resource, not on the cache itself.
Used across flows, APIs, and tools.
查找缓存是由Celigo管理的in-memory key-value store,用于在集成处理期间快速检索数据。与基于导出的查找(每条记录都要查询外部系统)不同,查找缓存可提供亚毫秒级的预加载参考数据访问,且不会消耗API调用次数。
查找缓存处理三类事项:
  • 缓存元数据——名称、集成范围,以及数据是否包含在模板/克隆中
  • 数据管理——通过专用数据端点加载、检索、更新和清除键值条目
  • 大小管控——每个缓存的容量限制为50 MB;每个环境下所有缓存的总容量限制为1 GB
查找缓存通过导入和导出映射中的
lookups[]
数组被调用。查找条目通过
_lookupCacheId
引用缓存,并可通过
extract
JSON路径从缓存对象中提取特定字段。此配置在导入/导出资源上完成,而非缓存本身。
适用于流程、API和各类工具。

"Lookup" -- Which One Is Meant?

“Lookup”——指代哪一种?

"Lookup" is overloaded in Celigo, and a lookup cache is only one thing the word can mean. When a request just says "the lookup," confirm which one before acting -- they live in different places and are configured differently. A single entry in
lookups[]
resolves its value one of three ways:
  • Cached -- the entry sets
    _lookupCacheId
    to read a lookup cache (this skill): pre-loaded reference data read in-memory, no per-record external call, as fresh as the last data load.
  • Static map -- the entry carries an inline
    map
    object. No cache and no external call; the table travels with the import/export. Best for a handful of stable pairs that rarely change.
  • Live / dynamic -- the entry issues a per-record query against a connected system (an HTTP request, SOQL, a NetSuite search, SQL). Always-fresh answers, at the cost of one external request per record.
Quick test for which is meant: pre-loaded data read in-memory is a lookup cache; an answer that must reflect the source system right now is a live/dynamic lookup; a small fixed table living on the step is a static
map
.
The same word also names things outside this skill: a mid-flow lookup step (an export with
isLookup: true
that fetches from an external system between flow steps) and transform lookups (tables inside an export transform's rules). "Lookup" alone is never enough -- listen for whether the subject is stored cache data, an inline rule on a step, a live query, or a separate flow step.
在Celigo中,“Lookup”一词有多种含义,查找缓存只是其中一种。当需求仅提到“lookup”时,需先确认具体指代哪一种——它们位于不同位置,配置方式也不同。
lookups[]
中的单个条目通过以下三种方式之一解析其值:
  • 缓存型——条目设置
    _lookupCacheId
    以读取查找缓存(即本技能):预加载的参考数据在内存中读取,无需每条记录调用外部系统,数据新鲜度与上次加载一致。
  • 静态映射型——条目携带内联
    map
    对象。无需缓存和外部调用;映射表随导入/导出一同存在。最适合少量稳定、极少变更的键值对。
  • 实时/动态型——条目针对连接的系统发起每条记录的查询(HTTP请求、SOQL、NetSuite搜索、SQL)。数据始终保持最新,但每条记录需消耗一次外部请求。
快速判断方法:预加载并在内存中读取的数据是查找缓存;必须实时反映源系统当前状态的是实时/动态lookup;存在于步骤中的小型固定表是静态
map
“Lookup”一词还指代本技能之外的内容:流程中间的lookup步骤(设置
isLookup: true
的导出步骤,用于在流程步骤之间从外部系统获取数据)和转换lookup(导出转换规则内的映射表)。仅用“Lookup”一词永远不够——需明确主题是存储的缓存数据、步骤上的内联规则、实时查询还是独立的流程步骤。

Use Cases

使用场景

Lookup caches serve several distinct patterns in production integrations:
查找缓存在生产集成中适用于多种不同模式:

Cross-Reference / ID Mapping

交叉引用/ID映射

The most common use case. Map external IDs to internal IDs across systems (e.g., Shopify product IDs to NetSuite internal IDs, external customer GUIDs to Salesforce Account IDs). Avoids repeated API lookups during import processing.
最常见的使用场景。在不同系统之间映射外部ID与内部ID(例如,Shopify产品ID与NetSuite内部ID、外部客户GUID与Salesforce Account ID)。避免在导入处理期间重复调用API进行查找。

Deduplication / Debouncing

去重/防抖

Track which records have already been processed to prevent duplicate operations. Store record IDs or composite keys as cache keys; check existence before processing.
跟踪已处理的记录,防止重复操作。将记录ID或复合键存储为缓存键;处理前检查键是否存在。

State Tracking

状态跟踪

Maintain timestamps, batch IDs, or processing markers across flow runs. Common patterns: "last sync time" caches, "open batch ID" holders, lock tables for concurrency control.
在多次流程运行之间维护时间戳、批次ID或处理标记。常见模式:“最后同步时间”缓存、“开放批次ID”存储、用于并发控制的锁表。

Static Reference Data

静态参考数据

Store relatively stable reference tables -- product catalogs, category hierarchies, shipping overrides, zip code lookups. Set
includeDataInTemplatesAndCloning: true
when this data is part of the integration's configuration.
存储相对稳定的参考表——产品目录、分类层级、运费覆盖规则、邮政编码查找表。当此类数据属于集成配置的一部分时,设置
includeDataInTemplatesAndCloning: true

Quick Reference

快速参考

Minimum Required Fields

必填字段

Every lookup cache needs at minimum:
name
.
Optional but common:
description
,
_integrationId
(scopes the cache to an integration),
includeDataInTemplatesAndCloning
.
每个查找缓存至少需要:
name
可选但常用的字段:
description
_integrationId
(将缓存限定到特定集成)、
includeDataInTemplatesAndCloning

Data Operations

数据操作

All data operations use the cache ID. Data entries are key-value pairs where both key and value are strings. Values can be JSON strings for complex objects -- the consuming lookup's
extract
JSON path pulls specific fields from the parsed value.
OperationMethodEndpointNotes
Upsert dataPOST
/v1/lookupcaches/{id}/data
Body:
{ "data": [{ "key": "k", "value": "v" }] }
Get dataPOST
/v1/lookupcaches/{id}/getData
No body = first page (max 1000). With
{ "keys": [...] }
or
{ "startsWith": "prefix" }
Delete keysDELETE
/v1/lookupcaches/{id}/data
Body:
{ "keys": ["k1", "k2"] }
Purge allDELETE
/v1/lookupcaches/{id}/data/purge
Removes all entries
所有数据操作都使用缓存ID。数据条目为键值对,键和值均为字符串。值可以是复杂对象的JSON字符串——调用缓存的lookup的
extract
JSON路径会从解析后的值中提取特定字段。
操作请求方法端点说明
插入/更新数据POST
/v1/lookupcaches/{id}/data
请求体:
{ "data": [{ "key": "k", "value": "v" }] }
获取数据POST
/v1/lookupcaches/{id}/getData
无请求体 = 获取第一页(最多1000条)。可携带
{ "keys": [...] }
{ "startsWith": "prefix" }
删除指定键DELETE
/v1/lookupcaches/{id}/data
请求体:
{ "keys": ["k1", "k2"] }
清除所有数据DELETE
/v1/lookupcaches/{id}/data/purge
删除所有条目

Schema Index

架构索引

All schemas are in references/schemas/:
  • Base fields (create/update): request.yml -- name, description, includeDataInTemplatesAndCloning, externalId
  • Response shape: response.yml -- includes size, sizeInMB, timestamps, integration/connector refs
所有架构均位于references/schemas/
  • 基础字段(创建/更新): request.yml ——名称、描述、includeDataInTemplatesAndCloning、externalId
  • 响应格式: response.yml ——包含大小、sizeInMB、时间戳、集成/连接器引用

Referencing Caches in Lookups

在Lookup中引用缓存

Lookup caches are consumed through the
lookups[]
array on imports and exports. See the writing-mappings skill for full lookup configuration details. The key fields:
  • name
    -- unique identifier for the lookup within the resource
  • _lookupCacheId
    -- references the cache resource
  • extract
    -- optional JSON path to pull a specific field from the cached value (e.g.,
    $.details.price
    )
  • default
    -- fallback value when the key is not found
  • allowFailures
    -- when
    true
    , missing keys use the default instead of failing the record
Reference the lookup by name in Handlebars expressions:
{{lookup 'lookupName' record.fieldName}}
.
查找缓存通过导入和导出上的
lookups[]
数组被调用。有关lookup的完整配置细节,请查看writing-mappings技能。关键字段:
  • name
    ——资源内lookup的唯一标识符
  • _lookupCacheId
    ——引用缓存资源
  • extract
    ——可选的JSON路径,用于从缓存值中提取特定字段(例如
    $.details.price
  • default
    ——键不存在时的 fallback 值
  • allowFailures
    ——设为
    true
    时,缺失的键会使用默认值,而非导致记录处理失败
在Handlebars表达式中通过名称引用lookup:
{{lookup 'lookupName' record.fieldName}}

Related Skills

相关技能

  • writing-mappings > Add lookups for value translation -- configuring
    lookups[]
    with
    _lookupCacheId
    ,
    extract
    , and
    allowFailures
  • writing-handlebars > Quick Reference --
    {{lookup}}
    helper syntax for referencing caches in templates
  • configuring-imports > How to Build an Import -- imports are the primary consumer of lookup caches
  • configuring-exports > Quick Reference -- lookup exports (
    isLookup: true
    ) as an alternative to caches for live data
  • building-flows > How to Build a Flow -- wiring lookups into flow pipelines
<!-- TIER:2 -->
  • writing-mappings > 添加用于值转换的lookup ——配置带有
    _lookupCacheId
    extract
    allowFailures
    lookups[]
  • writing-handlebars > 快速参考 ——在模板中引用缓存的
    {{lookup}}
    助手语法
  • configuring-imports > 如何构建导入 ——导入是查找缓存的主要调用方
  • configuring-exports > 快速参考 ——lookup导出(
    isLookup: true
    )作为缓存的替代方案,用于实时数据
  • building-flows > 如何构建流程 ——将lookup接入流程管道
<!-- TIER:2 -->

How to Build a Lookup Cache

如何构建查找缓存

1. Determine the use case

1. 确定使用场景

What data needs to be cached? Cross-reference IDs, static reference tables, deduplication keys, or processing state? This determines data structure, loading strategy, and lifecycle.
需要缓存哪些数据?交叉引用ID、静态参考表、去重键还是处理状态?这将决定数据结构、加载策略和生命周期。

2. Check for existing caches

2. 检查现有缓存

Before creating a new cache, search for existing ones in the account:
bash
undefined
创建新缓存前,先在账户中搜索现有缓存:
bash
undefined

List all lookup caches

列出所有查找缓存

celigo lookup-caches list
celigo lookup-caches list

Search across the account

在账户中搜索

celigo account search "lookup cache" celigo account search "<keyword related to your use case>"
undefined
celigo account search "lookup cache" celigo account search "<与你的使用场景相关的关键词>"
undefined

3. Create the cache

3. 创建缓存

Create the cache metadata first. Data is loaded separately.
bash
undefined
先创建缓存元数据,数据将单独加载。
bash
undefined

Create a new empty cache

创建新的空缓存

echo '{"name":"Product SKU to NetSuite ID","description":"Maps Shopify SKUs to NetSuite internal IDs for order import"}' | celigo lookup-caches create
echo '{"name":"Product SKU to NetSuite ID","description":"Maps Shopify SKUs to NetSuite internal IDs for order import"}' | celigo lookup-caches create

With integration scope and template inclusion

限定集成范围并包含到模板中

echo '{"name":"State Shipping Overrides","_integrationId":"<id>","includeDataInTemplatesAndCloning":true}' | celigo lookup-caches create
undefined
echo '{"name":"State Shipping Overrides","_integrationId":"<id>","includeDataInTemplatesAndCloning":true}' | celigo lookup-caches create
undefined

4. Load data into the cache

4. 向缓存加载数据

Use the
put-data
command to upsert key-value entries. The CLI auto-batches by count (1000 entries) and size (5 MB) per request.
bash
undefined
使用
put-data
命令插入/更新键值条目。CLI会自动按数量(1000条)和大小(5 MB)分批处理请求。
bash
undefined

Upsert entries

插入/更新条目

echo '{"data":[{"key":"SKU-001","value":"{"nsId":"12345","name":"Widget A"}"},{"key":"SKU-002","value":"{"nsId":"12346","name":"Widget B"}"}]}' | celigo lookup-caches put-data <cacheId>
echo '{"data":[{"key":"SKU-001","value":"{"nsId":"12345","name":"Widget A"}"},{"key":"SKU-002","value":"{"nsId":"12346","name":"Widget B"}"}]}' | celigo lookup-caches put-data <cacheId>

Simple string values for deduplication

用于去重的简单字符串值

echo '{"data":[{"key":"order-10001","value":"processed"},{"key":"order-10002","value":"processed"}]}' | celigo lookup-caches put-data <cacheId>

For large datasets, pipe from a script or file that generates the `{ "data": [...] }` JSON. The CLI handles batching automatically.
echo '{"data":[{"key":"order-10001","value":"processed"},{"key":"order-10002","value":"processed"}]}' | celigo lookup-caches put-data <cacheId>

对于大型数据集,可从生成`{ "data": [...] }` JSON的脚本或文件中管道输入。CLI会自动处理分批。

5. Verify the data

5. 验证数据

bash
undefined
bash
undefined

Get first page of all data (max 1000 keys)

获取所有数据的第一页(最多1000个键)

celigo lookup-caches get-data <cacheId>
celigo lookup-caches get-data <cacheId>

Get specific keys

获取特定键

echo '{"keys":["SKU-001","SKU-002"]}' | celigo lookup-caches get-data <cacheId>
echo '{"keys":["SKU-001","SKU-002"]}' | celigo lookup-caches get-data <cacheId>

Get keys by prefix

获取前缀匹配的键

echo '{"startsWith":"SKU-"}' | celigo lookup-caches get-data <cacheId>
echo '{"startsWith":"SKU-"}' | celigo lookup-caches get-data <cacheId>

Check cache size

检查缓存大小

celigo lookup-caches get <cacheId>
undefined
celigo lookup-caches get <cacheId>
undefined

6. Reference the cache in a lookup

6. 在Lookup中引用缓存

On the import or export resource, add an entry to
lookups[]
referencing the cache ID, then use the lookup name in field mappings or Handlebars expressions. See writing-mappings > Add lookups for value translation for the full configuration pattern.
在导入或导出资源上,向
lookups[]
添加引用缓存ID的条目,然后在字段映射或Handlebars表达式中使用lookup名称。完整配置模式请查看writing-mappings > 添加用于值转换的lookup

7. Plan the data refresh strategy

7. 规划数据刷新策略

Lookup caches are not automatically refreshed. Choose a strategy:
  • Manual -- update data via CLI or API as needed. Good for static reference tables.
  • Flow-driven -- use a scheduled flow with a postSubmit or preSavePage hook that calls the lookup cache data API to refresh entries. Good for cross-reference caches that need periodic sync.
  • Purge and reload -- purge all data and reload from scratch on a schedule. Good when the full dataset is small enough to reload quickly.
查找缓存不会自动刷新。选择合适的策略:
  • 手动刷新——根据需要通过CLI或API更新数据。适用于静态参考表。
  • 流程驱动刷新——使用带有postSubmit或preSavePage钩子的定时流程,调用查找缓存数据API刷新条目。适用于需要定期同步的交叉引用缓存。
  • 清除后重新加载——按计划清除所有数据并从头重新加载。适用于完整数据集较小、可快速重新加载的场景。

Purge vs Delete -- Data vs Resource

Purge与Delete——数据与资源的区别

Two different operations both sound like "clearing" or "removing" a cache. Choose by blast radius:
  • Purge empties the data (
    celigo lookup-caches purge-data <id>
    ) while the cache resource and every lookup that references its
    _lookupCacheId
    stay valid. After a purge, lookups just miss until data is reloaded, so their
    default
    /
    allowFailures
    behavior takes over. "Clear the cache" and "start over with fresh data" almost always mean purge.
  • Delete removes the resource itself (
    celigo lookup-caches delete <id>
    ). Every lookup entry pointing at that
    _lookupCacheId
    breaks. Reserve delete for "we don't use this cache anymore," and check what still references it first.
Purge is the reversible move -- reload restores the data and no references need rewiring, whereas a deleted cache's references all have to be repointed (and deleting is a soft delete -- see Gotchas).
Loads are upserts keyed by
key
: re-loading a refreshed dataset overwrites matching keys and adds new ones, but it does not remove keys that are absent from the new load. A true "replace the whole table" is therefore purge + reload, not a plain reload.
有两种操作听起来都像是“清除”或“移除”缓存。根据影响范围选择:
  • Purge 清空数据(
    celigo lookup-caches purge-data <id>
    ),但缓存资源和所有引用其
    _lookupCacheId
    的lookup仍保持有效。Purge后,lookup会返回缺失值,直到数据重新加载,因此它们的
    default
    /
    allowFailures
    行为会生效。“清除缓存”和“使用新数据重新开始”几乎总是指Purge。
  • Delete 移除资源本身(
    celigo lookup-caches delete <id>
    )。所有指向该
    _lookupCacheId
    的lookup条目都会失效。仅当“我们不再使用此缓存”时才使用Delete,且需先检查仍有哪些资源在引用它。
Purge是可撤销操作——重新加载即可恢复数据,无需重新配置引用;而删除缓存后,所有引用都必须重新指向(且删除是软删除——见注意事项)。
加载操作是基于
key
的插入/更新:重新加载刷新后的数据集会覆盖匹配的键并添加新键,但不会删除新数据集中不存在的键。因此,真正的“替换整个表”操作是Purge + 重新加载,而非单纯的重新加载。

CLI Commands

CLI命令

bash
undefined
bash
undefined

CRUD

CRUD操作

celigo lookup-caches list celigo lookup-caches get <id> celigo lookup-caches create < cache.json celigo lookup-caches update <id> < cache.json celigo lookup-caches set <id> key=value [key2=value2 ...] celigo lookup-caches delete <id> [-y]
celigo lookup-caches list celigo lookup-caches get <id> celigo lookup-caches create < cache.json celigo lookup-caches update <id> < cache.json celigo lookup-caches set <id> key=value [key2=value2 ...] celigo lookup-caches delete <id> [-y]

Data operations

数据操作

echo '{"data":[{"key":"k","value":"v"}]}' | celigo lookup-caches put-data <id> celigo lookup-caches get-data <id> # first page, all keys echo '{"keys":["k1","k2"]}' | celigo lookup-caches get-data <id> # specific keys echo '{"startsWith":"prefix"}' | celigo lookup-caches get-data <id> # prefix search echo '{"keys":["k1","k2"]}' | celigo lookup-caches delete-data <id> [-y] celigo lookup-caches purge-data <id> [-y]

<!-- TIER:3 -->
echo '{"data":[{"key":"k","value":"v"}]}' | celigo lookup-caches put-data <id> celigo lookup-caches get-data <id> # 获取第一页所有键 echo '{"keys":["k1","k2"]}' | celigo lookup-caches get-data <id> # 获取特定键 echo '{"startsWith":"prefix"}' | celigo lookup-caches get-data <id> # 前缀搜索 echo '{"keys":["k1","k2"]}' | celigo lookup-caches delete-data <id> [-y] celigo lookup-caches purge-data <id> [-y]

<!-- TIER:3 -->

Gotchas

注意事项

  1. PUT erases omitted fields. Always GET first, modify, then PUT. The
    set
    command handles this.
  2. Data upsert uses POST, not PUT.
    POST /v1/lookupcaches/{id}/data
    upserts entries. This is different from the resource-level PUT that updates cache metadata.
  3. Values must be strings. To store complex objects, JSON-stringify the value. The consuming lookup's
    extract
    path operates on the parsed JSON.
  4. 50 MB per cache, 1 GB per environment. Caches near the 50 MB limit (like the ~49 MB NetSuite Item caches seen in production) risk hitting the ceiling on the next upsert. Monitor
    size
    and
    sizeInMB
    on the response.
  5. get-data
    returns max 1000 keys per call.
    For caches with more than 1000 entries, use
    startsWith
    prefix queries or specific key lookups to retrieve data.
  6. No automatic TTL or expiry. Lookup cache data persists until explicitly deleted or purged. Stale data is a common source of bugs -- plan a refresh strategy.
  7. includeDataInTemplatesAndCloning
    defaults to false.
    If the cache data is part of the integration's configuration (static reference tables, shipping overrides), set this to
    true
    or the data will be lost when cloning or installing from a template.
  8. Sandbox and production caches are separate. A
    sandbox: true
    cache is only accessible to sandbox flows. Production flows cannot read sandbox caches and vice versa.
  9. Deleting a cache is a soft delete. The cache is retained for 30 days before permanent removal. During this window, a cache with the same name cannot be re-created with the same
    _id
    .
  1. PUT会删除未指定的字段。始终先GET,修改后再PUT。
    set
    命令会自动处理此问题。
  2. 数据插入/更新使用POST而非PUT
    POST /v1/lookupcaches/{id}/data
    用于插入/更新条目。这与更新缓存元数据的资源级PUT不同。
  3. 值必须是字符串。要存储复杂对象,需将值JSON序列化。调用缓存的lookup的
    extract
    路径会在解析后的JSON上操作。
  4. 每个缓存50 MB,每个环境1 GB。接近50 MB限制的缓存(例如生产环境中约49 MB的NetSuite Item缓存)在下次插入/更新时可能会达到上限。监控响应中的
    size
    sizeInMB
    字段。
  5. get-data
    每次调用最多返回1000个键
    。对于包含超过1000个条目的缓存,使用
    startsWith
    前缀查询或特定键查找来检索数据。
  6. 无自动TTL或过期机制。查找缓存数据会一直保留,直到被显式删除或Purge。过期数据是常见的bug来源——需规划刷新策略。
  7. includeDataInTemplatesAndCloning
    默认值为false
    。如果缓存数据属于集成配置的一部分(静态参考表、运费覆盖规则),请将其设为
    true
    ,否则在克隆或从模板安装时数据会丢失。
  8. 沙箱与生产环境的缓存相互独立
    sandbox: true
    的缓存仅对沙箱流程可见。生产流程无法读取沙箱缓存,反之亦然。
  9. 删除缓存是软删除。缓存会保留30天后才被永久删除。在此期间,无法使用相同的
    _id
    创建同名缓存。

Safe Lookup-to-Update -- Never Write Off an Unguarded First Match

安全的Lookup到更新流程——绝不要基于无防护的首个匹配进行写入

Cross-reference resolution -- the flagship cache use case -- resolves a key (an internal ID, an account ID) that then feeds a downstream update or write. A cache read returns exactly one value per key, so resolving on a genuinely unique key (an external ID, a primary key) is safe by construction. The danger lives in any lookup that can return more than one match -- typically a live/dynamic lookup on a fuzzy or human key (name, email, phone number), where duplicates are normal.
Feeding the first result (
data[0]
) of a multi-match-capable lookup into an update writes an arbitrary wrong record on every duplicate -- a silent data-corruption bug, not a skipped record, and one wrong-record update is worse than a skipped record. Never feed, skip on, or branch on an unguarded first match. Guard it one of two ways:
  • Configure the resolving lookup to fail or skip the record on multiple matches, so a duplicate never reaches the write, OR
  • Check that the result count equals one before the write -- an input filter on the writing step, or a
    postResponseMap
    hook that drops or errors records whose lookup returned more than one result.
Make the guard explicit when you build a lookup-then-update: a bare
data[0]
mapping paired with an assumption that the lookup "should" return a single match is exactly the shape that corrupts data in production.
交叉引用解析——缓存的核心使用场景——解析一个键(内部ID、账户ID),然后将其用于下游更新或写入。缓存读取每个键仅返回一个值,因此基于真正唯一的键(外部ID、主键)进行解析本质上是安全的。风险存在于任何可能返回多个匹配结果的lookup中——通常是基于模糊或人工键(名称、邮箱、电话号码)的实时/动态lookup,重复数据是正常现象。
将多匹配lookup的首个结果(
data[0]
)用于下游更新,会在每次遇到重复数据时写入任意错误的记录——这是一个静默的数据损坏bug,而非跳过记录,且一次错误记录更新比跳过记录更严重。绝不要基于无防护的首个匹配进行写入、跳过或分支处理。可通过以下两种方式防护:
  • 将解析lookup配置为在多匹配时失败或跳过记录,使重复数据永远无法到达写入步骤,OR
  • 在写入前检查结果数量是否为1——在写入步骤添加输入过滤器,或使用
    postResponseMap
    钩子丢弃或标记lookup返回多个结果的记录为错误。
构建lookup后更新流程时,请明确添加防护措施:仅使用
data[0]
映射并假设lookup“应该”返回单个结果的配置,正是会在生产环境中导致数据损坏的典型情况。

Common Errors

常见错误

ErrorCauseFix
422 on data upsertEntries exceed 5 MB batch limit or individual entry too largeReduce batch size; the CLI auto-batches at 1000 entries / 5 MB
404 on data operationsCache ID does not exist or was deletedVerify the cache ID with
celigo lookup-caches get <id>
Lookup returns
null
in flow
Key does not exist in the cacheCheck that the cache is populated and the key format matches exactly (case-sensitive)
Lookup returns full object instead of fieldMissing
extract
path on the lookup definition
Add
extract: "$.fieldName"
to the lookup entry on the import/export
403 on cache operationsAccount does not have the Lookup Cache licenseContact Celigo to enable the lookup cache feature
Cache size approaching 50 MBToo much data for a single cachePurge stale entries, split into multiple caches by category, or archive old data
Downstream update modifies the wrong recordThe first result (
data[0]
) of a multi-match-capable lookup was fed into a write with no single-match guard
Configure the lookup to fail/skip on multiple matches, or verify the result count equals one before writing
"Clearing" a cache breaks every lookup that used itThe cache resource was deleted instead of having its data purgedUse
purge-data
to empty entries while keeping references valid; delete only to retire the cache entirely
错误原因修复方案
数据插入/更新时返回422条目超过5 MB批次限制或单个条目过大减小批次大小;CLI会自动按1000条/5 MB分批
数据操作时返回404缓存ID不存在或已被删除使用
celigo lookup-caches get <id>
验证缓存ID
流程中Lookup返回
null
键不存在于缓存中检查缓存是否已填充,且键格式完全匹配(区分大小写)
Lookup返回完整对象而非指定字段lookup定义中缺少
extract
路径
在导入/导出的lookup条目中添加
extract: "$.fieldName"
缓存操作时返回403账户没有Lookup Cache许可证联系Celigo启用查找缓存功能
缓存大小接近50 MB单个缓存存储的数据过多Purge过期条目,按类别拆分为多个缓存,或归档旧数据
下游更新修改了错误记录将多匹配lookup的首个结果(
data[0]
)用于写入,且未添加单匹配防护
将lookup配置为在多匹配时失败/跳过,或在写入前验证结果数量为1
“清除”缓存后所有使用它的lookup都失效了删除了缓存资源而非清除其数据使用
purge-data
清空条目并保持引用有效;仅在完全停用缓存时才使用delete