mantis-structural-index

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

Structural Code Index Builder

结构代码索引构建器

This is an optional first-class stage in the Pass Lifecycle Contract. It runs immediately after the snapshot is pinned (Block D), before the first code-reading analysis stage (summarize/architecture). It only needs
CODE_ROOT
+
SNAPSHOT_ID
and must not depend on architecture/KB.
这是Pass Lifecycle Contract中的一个可选一级阶段。它会在快照固定完成后(Block D)立即运行,早于首个代码读取分析阶段(摘要/架构分析)。它仅需
CODE_ROOT
+
SNAPSHOT_ID
,且不得依赖架构/知识库(KB)。

System Goal

系统目标

Structural Code Index Builder. Builds a content-addressed semantic-unit index from source code using capability-based per-partition backend selection, degrading gracefully to grep. Provides
find_callers(symbol)
,
get_function_boundary(file, line)
, and call-site awareness to improve LLM reasoning quality during discovery — supplementing (never replacing) grep-based call-site discovery with structural data. The index is composed from independent semantic compilation units, persisted as a manifest + SQLite catalog, and exposed through a bounded query helper.
结构代码索引构建器。基于能力的分区后端选择机制,从源代码构建基于内容寻址的语义单元索引,当无法使用更优方案时优雅降级为grep。提供
find_callers(symbol)
get_function_boundary(file, line)
方法以及调用站点感知能力,以提升发现阶段的LLM推理质量——用结构化数据补充(绝不替代)基于grep的调用站点发现。该索引由独立的语义编译单元组成,以清单(manifest)+ SQLite目录形式持久化,并通过受限查询助手对外暴露。

Command Definition

命令定义

  • Command:
    /mantis-structural-index
  • Description: Build a content-addressed semantic-unit index from source code under
    CODE_ROOT
    .
  • Arguments (optional; supplied by the orchestrator, consumed by Block A):
    --snapshot_root
    /
    --snapshot_id
    /
    --state_root
    . All absent → MODE-OFF/legacy mode (reads source from the current directory, writes index to
    ./workspace/kb/structural_index/
    ).
  • 命令:
    /mantis-structural-index
  • 描述:
    CODE_ROOT
    下的源代码构建基于内容寻址的语义单元索引。
  • 参数(可选;由编排器提供,供Block A使用):
    --snapshot_root
    /
    --snapshot_id
    /
    --state_root
    。若所有参数均缺失→进入MODE-OFF/传统模式(从当前目录读取源代码,将索引写入
    ./workspace/kb/structural_index/
    )。

Input/Output Contract

输入/输出约定

  • Reads:
    • workspace/.mantis_state.json
      (to read
      active_snapshot
      for provenance checking and snapshot-aware rebuild logic).
    • workspace/kb/structural_index/manifest.json
      (to check
      snapshot_id
      for reuse-on-match idempotency — primary check).
    • workspace/kb/structural_index.jsonl
      (backward-compat provenance check if manifest absent).
    • workspace/kb/structural_index/units/
      (content-addressed cache for incremental unit reuse).
    • workspace/kb/structural_index/native/
      and sidecar
      provenance.json
      files (prebuilt index attachments and metadata manifests).
    • CODE_ROOT source files (via generated helper script — the script parses all source files under
      CODE_ROOT
      ).
  • Writes:
    • workspace/kb/structural_index/manifest.json
      (STATE-RELATIVE — atomic commit point, written LAST).
    • workspace/kb/structural_index/catalog.sqlite
      (STATE-RELATIVE — query-optimized serving store).
    • workspace/kb/structural_index/units/
      (STATE-RELATIVE — content-addressed immutable unit outputs).
    • workspace/kb/structural_index/shards/
      (STATE-RELATIVE — partitioned serving data for large corpora).
    • workspace/kb/structural_index/native/
      (STATE-RELATIVE — prebuilt index attachments: SCIP, Kythe, LSIF).
    • workspace/kb/structural_index/tmp/
      (STATE-RELATIVE — temporary objects during build).
    • workspace/kb/structural_index.jsonl
      (STATE-RELATIVE — compatibility pointer; full export below threshold).
    • workspace/helpers/build_structural_index.py
      (STATE-RELATIVE — the builder helper script).
    • workspace/helpers/query_structural_index.py
      (STATE-RELATIVE — the query helper script).
  • Preconditions:
    • Source files must exist under
      CODE_ROOT
      . If
      CODE_ROOT
      is not resolved (MODE-OFF and no readable
      active_snapshot
      ), build against the current directory with
      snapshot_id
      set to
      "unknown"
      . Do NOT skip — this is the standalone-efficiency case.
  • Inert until wired: This skill returns an empty index until a caller invokes it (the harness,
    mantis-plan
    , or
    mantis-researcher
    ). It never fails — it simply returns an empty index if tools are unavailable or source cannot be parsed.
  • Idempotency Guarantee:
    • Read-only on
      CODE_ROOT
      . Writes only to STATE-RELATIVE paths. Re-running with the same
      CODE_ROOT
      and
      SNAPSHOT_ID
      reuses the existing index (manifest
      snapshot_id
      match) rather than rebuilding — except in MODE-OFF, where it always rebuilds. Individual semantic units are reused across snapshots when their content-addressed cache keys match (incremental rebuild).
  • 读取:
    • workspace/.mantis_state.json
      (读取
      active_snapshot
      以进行溯源检查和快照感知的重建逻辑)。
    • workspace/kb/structural_index/manifest.json
      (检查
      snapshot_id
      以实现匹配时复用的幂等性——主要检查项)。
    • workspace/kb/structural_index.jsonl
      (若清单缺失,用于向后兼容的溯源检查)。
    • workspace/kb/structural_index/units/
      (基于内容寻址的缓存,用于增量单元复用)。
    • workspace/kb/structural_index/native/
      及附属
      provenance.json
      文件(预构建的索引附件和元数据清单)。
    • CODE_ROOT下的源代码文件(通过生成的助手脚本读取——该脚本解析CODE_ROOT下所有源代码文件)。
  • 写入:
    • workspace/kb/structural_index/manifest.json
      (与状态相关——原子提交点,最后写入)。
    • workspace/kb/structural_index/catalog.sqlite
      (与状态相关——查询优化的服务存储)。
    • workspace/kb/structural_index/units/
      (与状态相关——基于内容寻址的不可变单元输出)。
    • workspace/kb/structural_index/shards/
      (与状态相关——针对大型语料库的分区服务数据)。
    • workspace/kb/structural_index/native/
      (与状态相关——预构建的索引附件:SCIP、Kythe、LSIF)。
    • workspace/kb/structural_index/tmp/
      (与状态相关——构建过程中的临时对象)。
    • workspace/kb/structural_index.jsonl
      (与状态相关——兼容指针;低于阈值时导出完整内容)。
    • workspace/helpers/build_structural_index.py
      (与状态相关——构建助手脚本)。
    • workspace/helpers/query_structural_index.py
      (与状态相关——查询助手脚本)。
  • 前置条件:
    • CODE_ROOT下必须存在源代码文件。若CODE_ROOT未解析(MODE-OFF且无可读的
      active_snapshot
      ),则针对当前目录构建,
      snapshot_id
      设为
      "unknown"
      。请勿跳过——这是独立高效的使用场景。
  • 未触发时无操作: 该技能在被调用前(由 harness、
    mantis-plan
    mantis-researcher
    调用)会返回空索引。它永远不会失败——若工具不可用或源代码无法解析,仅返回空索引。
  • 幂等性保证:
    • 对CODE_ROOT仅执行读取操作。仅写入与状态相关的路径。若使用相同的CODE_ROOT和SNAPSHOT_ID重新运行,会复用现有索引(清单
      snapshot_id
      匹配)而非重建——但MODE-OFF模式除外,该模式下始终重建。当语义单元的内容寻址缓存键匹配时,可跨快照复用单个语义单元(增量重建)。

Instructions

操作步骤

Step 0: Locator Resolution (run first)

步骤0:定位器解析(首先运行)

LOCATOR RESOLUTION (before reading ANY target code or artifact):
0. ROLE: If this skill NEVER reads target source (report, calibrate, reflect),
   you are a FINDINGS-ONLY stage: skip steps 2-6; still read active_snapshot from
   state for provenance/annotation; NEVER stop merely because a code root is unset.
1. Determine CODE_ROOT, in this priority order:
   a. If --target_root is passed on THIS invocation, CODE_ROOT = --target_root.
      It is AUTHORITATIVE and OVERRIDES SNAPSHOT_ROOT and the state fallback
      (used when a caller hands you a prepared tree, e.g. a patched shadow).
   b. Else if --snapshot_root (or SNAPSHOT_ROOT) is passed, use it.
   c. Else read state_root/workspace/.mantis_state.json (state_root from
      --state_root if passed, else ./workspace/... relative to the current dir)
      -> active_snapshot.root / .snapshot_id / .snapshot_pinned.
   d. Else (no arg AND no readable active_snapshot): CODE_ROOT = current directory,
      treat snapshot_pinned = false (MODE-OFF). Do NOT stop.
2. SENTINEL CHECK (only if snapshot_pinned is true AND you did NOT take path 1a):
   verify CODE_ROOT/.mantis_snapshot_id exists and equals SNAPSHOT_ID. If missing
   or different -> STOP "snapshot sentinel mismatch". (A --target_root tree (1a) is
   deliberately mutated and is sentinel-EXEMPT.)
3. PATH FIELDS:
   - SNAPSHOT-RELATIVE (read under CODE_ROOT): code_paths entries; plan target_files
     that are file paths. Strip ONLY a trailing ":<digits>". A code_paths entry
     containing "://" is a URL/endpoint, NOT a file read. A code_paths entry that is
     NOT of the form <existing-path>:<integer> is a non-source LOCATOR
     (symbol/offset/endpoint): only check that the artifact/symbol exists; skip ALL
     line-range and line-existence logic.
   - STATE-RELATIVE (read/write under state_root/workspace, NEVER prefix CODE_ROOT):
     kb_references, repro_file_path, reattack_file_path, helper scripts, report
     files, and all state/findings JSON.
4. Never WRITE under CODE_ROOT when snapshot_pinned is true. Any command that
   compiles, generates, or writes artifacts MUST run in a PRIVATE SHADOW copy
   (mktemp -d from CODE_ROOT), never with cwd=CODE_ROOT. Read-only inspection may
   cd into CODE_ROOT.
5. VCS-METADATA CARVE-OUT: history-log extraction and any VCS diff/blame command
   run in the LIVE repository root (which still has .git/.hg/.repo), NOT CODE_ROOT
   (the snapshot copy strips VCS metadata). Do NOT stop merely because CODE_ROOT
   lacks .git/.hg/.repo.
6. Every shell command uses ABSOLUTE paths and sets its own working directory on
   that call. Do NOT assume the working directory persists between calls.
This is a CODE-READING stage — it reads target source files under CODE_ROOT via the helper script. Block A step 0's findings-only skip does NOT apply.
LOCATOR RESOLUTION (before reading ANY target code or artifact):
0. ROLE: If this skill NEVER reads target source (report, calibrate, reflect),
   you are a FINDINGS-ONLY stage: skip steps 2-6; still read active_snapshot from
   state for provenance/annotation; NEVER stop merely because a code root is unset.
1. Determine CODE_ROOT, in this priority order:
   a. If --target_root is passed on THIS invocation, CODE_ROOT = --target_root.
      It is AUTHORITATIVE and OVERRIDES SNAPSHOT_ROOT and the state fallback
      (used when a caller hands you a prepared tree, e.g. a patched shadow).
   b. Else if --snapshot_root (or SNAPSHOT_ROOT) is passed, use it.
   c. Else read state_root/workspace/.mantis_state.json (state_root from
      --state_root if passed, else ./workspace/... relative to the current dir)
      -> active_snapshot.root / .snapshot_id / .snapshot_pinned.
   d. Else (no arg AND no readable active_snapshot): CODE_ROOT = current directory,
      treat snapshot_pinned = false (MODE-OFF). Do NOT stop.
2. SENTINEL CHECK (only if snapshot_pinned is true AND you did NOT take path 1a):
   verify CODE_ROOT/.mantis_snapshot_id exists and equals SNAPSHOT_ID. If missing
   or different -> STOP "snapshot sentinel mismatch". (A --target_root tree (1a) is
   deliberately mutated and is sentinel-EXEMPT.)
3. PATH FIELDS:
   - SNAPSHOT-RELATIVE (read under CODE_ROOT): code_paths entries; plan target_files
     that are file paths. Strip ONLY a trailing ":<digits>". A code_paths entry
     containing "://" is a URL/endpoint, NOT a file read. A code_paths entry that is
     NOT of the form <existing-path>:<integer> is a non-source LOCATOR
     (symbol/offset/endpoint): only check that the artifact/symbol exists; skip ALL
     line-range and line-existence logic.
   - STATE-RELATIVE (read/write under state_root/workspace, NEVER prefix CODE_ROOT):
     kb_references, repro_file_path, reattack_file_path, helper scripts, report
     files, and all state/findings JSON.
4. Never WRITE under CODE_ROOT when snapshot_pinned is true. Any command that
   compiles, generates, or writes artifacts MUST run in a PRIVATE SHADOW copy
   (mktemp -d from CODE_ROOT), never with cwd=CODE_ROOT. Read-only inspection may
   cd into CODE_ROOT.
5. VCS-METADATA CARVE-OUT: history-log extraction and any VCS diff/blame command
   run in the LIVE repository root (which still has .git/.hg/.repo), NOT CODE_ROOT
   (the snapshot copy strips VCS metadata). Do NOT stop merely because CODE_ROOT
   lacks .git/.hg/.repo.
6. Every shell command uses ABSOLUTE paths and sets its own working directory on
   that call. Do NOT assume the working directory persists between calls.
这是一个代码读取阶段——它通过助手脚本读取CODE_ROOT下的目标源代码。Block A步骤0中的仅发现结果跳过规则不适用于此阶段。

Step 1: Idempotency / Freshness Check (MANDATORY FIRST)

步骤1:幂等性/新鲜度检查(必须首先执行)

MANDATORY FIRST STEP: Before writing any build scripts, probing backends, or extracting symbols, check for MODE-OFF or an existing manifest:
  1. MODE-OFF check (FIRST): In MODE-OFF (
    SNAPSHOT_ID
    is
    "unknown"
    or absent) → always rebuild. Do NOT reuse a previous
    "unknown"
    index, because the live tree is mutable and
    "unknown"
    is a constant (not a freshness signal). Skip directly to Step 2 (individual units whose content has not changed may still hit the content-addressed cache).
  2. If
    workspace/kb/structural_index/manifest.json
    exists, read its
    snapshot_id
    field.
  3. If
    snapshot_id
    matches the current
    SNAPSHOT_ID
    (and
    SNAPSHOT_ID
    is NOT
    "unknown"
    ) → reuse the index immediately and STOP. Do NOT invoke Step 2, probe backends, or write build scripts. This bounds cost across retries/crash-resume.
  4. If
    manifest.json
    is absent but
    workspace/kb/structural_index.jsonl
    exists (backward compat), read its provenance header (
    _provenance
    ,
    snapshot_id
    keys). If
    snapshot_id
    matches (and is NOT
    "unknown"
    ) → reuse and STOP. Otherwise proceed to rebuild.
  5. If
    SNAPSHOT_ID
    differs → proceed to rebuild (Steps 2–5). For incremental reuse: before rebuilding a semantic unit, check
    units/
    for a content-addressed cache hit (see Content-Addressed Cache Key below). A cache hit reuses the unit output without re-extraction.
必须首先执行的步骤: 在编写任何构建脚本、探测后端或提取符号之前,检查是否处于MODE-OFF模式或存在现有清单:
  1. MODE-OFF检查(首先执行): 若处于MODE-OFF模式(
    SNAPSHOT_ID
    "unknown"
    或缺失)→ 始终重建。请勿复用之前的
    "unknown"
    索引,因为实时代码树是可变的,
    "unknown"
    是常量(而非新鲜度信号)。直接跳至步骤2(内容未更改的单个单元仍可命中基于内容寻址的缓存)。
  2. workspace/kb/structural_index/manifest.json
    存在,读取其
    snapshot_id
    字段。
  3. snapshot_id
    与当前
    SNAPSHOT_ID
    匹配(且
    SNAPSHOT_ID
    不为
    "unknown"
    )→ 立即复用索引并停止操作。请勿执行步骤2、探测后端或编写构建脚本。这可限制重试/崩溃恢复场景下的成本。
  4. manifest.json
    缺失但
    workspace/kb/structural_index.jsonl
    存在(向后兼容),读取其溯源头(
    _provenance
    snapshot_id
    键)。若
    snapshot_id
    匹配(且不为
    "unknown"
    )→ 复用并停止操作。否则继续重建。
  5. SNAPSHOT_ID
    不同→继续重建(步骤2–5)。对于增量复用:重建语义单元前,检查
    units/
    是否存在内容寻址缓存命中(见下文基于内容寻址的缓存键)。缓存命中时可复用单元输出,无需重新提取。

Step 2: Select Backend per Partition

步骤2:按分区选择后端

Partition the codebase into semantic units (see Per-Language Semantic Units below). For each unit, select the most precise backend available in this environment using a capability-based, per-partition decision — not a single global ladder. The following are examples, not an exhaustive enum. The helper probes each tier per partition and selects the highest available:
  1. Snapshot-matched SCIP / LSIF / Kythe / clangd-static / remote index (precision:
    semantic
    ) — if a pre-built index matching the current
    snapshot_id
    or
    root_fingerprint
    is available. Most precise: full type-aware cross-reference, call hierarchy, and hover/signature data.
  2. Compiler / typechecker-backed extractor (precision:
    typecheck
    ) — if
    compile_commands.json
    , build context, or typechecker is available. Type-accurate symbol resolution and call edges.
  3. Language-aware AST extraction (precision:
    ast
    ) — if
    tree-sitter
    ,
    ast-grep
    , or a language-specific parser is available. Full AST parsing: function boundaries, call expressions, signatures.
  4. Symbol-only extraction (precision:
    symbol-only
    ) — if
    ctags
    or equivalent is on
    PATH
    . Symbol table only (function definitions, locations — no call graph). Call-site extraction uses a lightweight regex pass within known function boundaries.
  5. Heuristic fallback (precision:
    heuristic
    ) — Python stdlib regex pass over source files. Identifies function definitions and call patterns using language-agnostic heuristics. Less precise but zero-dependency.
  6. Coverage-only manifest + lexical fallback (precision:
    coverage-only
    ) — grep; no structural index is written. Manifest records
    status: "empty"
    . Consumers fall back to grep-based discovery (today's behavior byte-for-byte).
Native Index Probing & Resolution Rules:
  • Probe instruction: Before evaluating per-partition backends, probe
    workspace/kb/structural_index/native/
    and subdirectories
    native/{scip,lsif,kythe}/
    for prebuilt index files.
  • Snapshot-declaration convention: Because native formats (SCIP, LSIF, Kythe) do not embed snapshot identity directly in their binary payload, prebuilt indexes MUST declare their target snapshot using a sidecar
    provenance.json
    manifest located at
    workspace/kb/structural_index/native/provenance.json
    or
    native/<kind>/provenance.json
    . The manifest contains an array of attachments:
    [{"kind": "scip|lsif|kythe", "path": "...", "snapshot_id": "...", "root_fingerprint": "...", "language": "...", "indexer": "...", "precision": "semantic", "files": [...]}]
    . A native index is matched if its declared
    snapshot_id
    equals
    SNAPSHOT_ID
    (when
    SNAPSHOT_ID != "unknown"
    ) or its
    root_fingerprint
    matches the workspace's calculated root fingerprint. If the
    files
    array is absent or empty, treat the native index as covering no individual files directly (record in
    manifest.native_indexes
    but do not update
    coverage
    rows; fall through to lower tiers for all files).
  • Record-and-Defer Ingestion Rule: Parsing raw binary native indexes (e.g. SCIP protobuf) in pure Python without dependencies is costly and complex. Option A builder scripts MUST detect matching prebuilt native indexes, record their entries in
    manifest.native_indexes
    , and set the
    coverage
    table
    backend
    (e.g.,
    "scip"
    or
    "scip-clangd"
    ) for all files listed in the provenance manifest. If
    catalog.sqlite
    is NOT populated with symbols from the native index (raw binary deferred to harness/MCP readers), set
    coverage.status = "deferred"
    and
    precision = "deferred"
    (with
    indexed_files = 0
    ). This prevents the query helper from claiming an un-ingested partition is "authoritative empty" at
    semantic
    precision, ensuring consumers run the mandatory grep fallback. When
    catalog.sqlite
    IS populated (e.g., via Option B pre-ingestion or
    scip-to-sqlite
    ), set
    precision = "semantic"
    and
    status = "indexed"
    .
LSP is NOT equivalent to SCIP / LSIF. LSP is an interactive protocol whose workspace state may be partial or mutable. Use it only when the server can demonstrate snapshot identity AND complete workspace coverage. A running language server does not automatically qualify as a
semantic
backend.
SCIP merging. SCIP explicitly permits merging complementary information from indexers with different precision levels. Its format records indexer version and per-document language metadata. The builder MAY merge results from multiple indexers (e.g., a SCIP index for Go + tree-sitter for Python) within a single catalog.
The determinism lives in a runtime-generated versioned helper (
build_structural_index.py
,
# MANTIS_HELPER_VERSION = 5
, grep-and-regenerate on reuse) that probes and selects backends per partition. No shipped binaries; air-gapped-safe.
Deterministic partial coverage. On very large source trees, a full rebuild can dominate stage-0 wall-clock. Replace discovery-order truncation with a deterministic priority queue:
  1. Explicit target files and symbols (from
    plan.json
    , if available).
  2. Changed units and known reverse dependencies (from Block E diff + reverse-dependency edges in the existing catalog).
  3. Containing packages/modules and direct imports of tier 1+2 units.
  4. Remaining units in normalized path order.
Apply deterministic
max_units
(default: 10000) or
max_source_bytes
(default: 500MB) bounds to that queue. Persist deferred units in the manifest so another invocation can resume. Statuses:
complete
,
partial
,
empty
,
failed
.
A consumer should only interpret "no callers" strongly when the relevant partition is complete, snapshot-matched, covered, and semantically precise. Otherwise the answer is "no indexed callers", followed by the existing lexical fallback. The index is HINT-only, so partial coverage is safe (grep remains authoritative).
将代码库划分为语义单元(见下文按语言划分的语义单元)。针对每个单元,使用基于能力的分区决策选择当前环境中可用的最精准后端——而非单一全局优先级。以下是示例,而非详尽枚举。助手会针对每个分区探测各层级并选择最高可用层级:
  1. 快照匹配的SCIP / LSIF / Kythe / clangd-static / 远程索引(精度:
    semantic
    )——若存在与当前
    snapshot_id
    root_fingerprint
    匹配的预构建索引。精度最高:支持完整的类型感知交叉引用、调用层级以及悬停/签名数据。
  2. 编译器/类型检查器驱动的提取器(精度:
    typecheck
    )——若存在
    compile_commands.json
    、构建上下文或类型检查器。可实现类型准确的符号解析和调用边。
  3. 感知语言的AST提取(精度:
    ast
    )——若
    tree-sitter
    ast-grep
    或特定语言解析器可用。支持完整AST解析:函数边界、调用表达式、签名。
  4. 仅符号提取(精度:
    symbol-only
    )——若
    ctags
    或等效工具在
    PATH
    中。仅提供符号表(函数定义、位置——无调用图)。调用站点提取通过已知函数边界内的轻量级正则扫描实现。
  5. 启发式回退(精度:
    heuristic
    )——通过Python标准库正则扫描源代码文件。使用语言无关的启发式方法识别函数定义和调用模式。精度较低但无依赖。
  6. 仅覆盖清单+词法回退(精度:
    coverage-only
    )——仅使用grep;不写入结构索引。清单记录
    status: "empty"
    。消费者回退到基于grep的发现(与当前行为完全一致)。
原生索引探测与解析规则:
  • 探测指令: 在评估分区后端之前,探测
    workspace/kb/structural_index/native/
    及其子目录
    native/{scip,lsif,kythe}/
    以查找预构建索引文件。
  • 快照声明约定: 由于原生格式(SCIP、LSIF、Kythe)未在其二进制负载中直接嵌入快照标识,预构建索引必须通过附属的
    provenance.json
    清单声明其目标快照,该清单位于
    workspace/kb/structural_index/native/provenance.json
    native/<kind>/provenance.json
    。清单包含一个附件数组:
    [{"kind": "scip|lsif|kythe", "path": "...", "snapshot_id": "...", "root_fingerprint": "...", "language": "...", "indexer": "...", "precision": "semantic", "files": [...]}]
    。若原生索引声明的
    snapshot_id
    等于
    SNAPSHOT_ID
    (当
    SNAPSHOT_ID != "unknown"
    时),或其
    root_fingerprint
    与工作区计算的根指纹匹配,则该索引匹配。若
    files
    数组缺失或为空,则认为原生索引不直接覆盖任何单个文件(在
    manifest.native_indexes
    中记录,但不更新
    coverage
    行;所有文件均回退到更低层级)。
  • 记录并延迟摄取规则: 在纯Python环境中无依赖地解析原始二进制原生索引(如SCIP protobuf)成本高昂且复杂。Option A构建脚本必须检测匹配的预构建原生索引,在
    manifest.native_indexes
    中记录其条目,并为溯源清单中列出的所有文件设置
    coverage
    表的
    backend
    (例如
    "scip"
    "scip-clangd"
    )。若
    catalog.sqlite
    未填充来自原生索引的符号(原始二进制延迟到harness/MCP读取器处理),则设置
    coverage.status = "deferred"
    precision = "deferred"
    indexed_files = 0
    )。这可防止查询助手声称未摄取的分区在
    semantic
    精度下“权威为空”,确保消费者执行强制grep回退。若
    catalog.sqlite
    已填充(例如通过Option B预摄取或
    scip-to-sqlite
    ),则设置
    precision = "semantic"
    status = "indexed"
LSP不等同于SCIP / LSIF。 LSP是交互式协议,其工作区状态可能不完整或可变。仅当服务器可证明快照标识且覆盖完整工作区时,才可使用LSP。运行中的语言服务器并不自动符合
semantic
后端的要求。
SCIP合并。 SCIP明确允许合并来自不同精度级别索引器的互补信息。其格式记录了索引器版本和每个文档的语言元数据。构建器可在单个目录中合并来自多个索引器的结果(例如Go的SCIP索引 + Python的tree-sitter索引)。
确定性体现在运行时生成的版本化助手(
build_structural_index.py
# MANTIS_HELPER_VERSION = 5
,复用前通过grep检查并重新生成)中,该助手会探测并按分区选择后端。无预装二进制文件;支持离线环境。
确定性部分覆盖。 在超大型源代码树上,完整重建可能占据阶段0的主要耗时。将发现顺序截断替换为确定性优先级队列:
  1. 明确的目标文件和符号(若
    plan.json
    可用)。
  2. 已更改的单元和已知反向依赖(来自Block E差异 + 现有目录中的反向依赖边)。
  3. 包含的包/模块以及一级+二级单元的直接导入
  4. 剩余单元按标准化路径顺序排列
为该队列应用确定性的
max_units
(默认:10000)或
max_source_bytes
(默认:500MB)限制。将延迟处理的单元持久化到清单中,以便下次调用可继续。状态包括:
complete
partial
empty
failed
仅当相关分区已完成、快照匹配、已覆盖且语义精准时,消费者才可将“无调用者”的结果视为可靠。否则结果应为“无索引调用者”,随后执行现有词法回退。索引仅作为提示,因此部分覆盖是安全的(grep仍为权威来源)。

Step 3: Write and Run Helper Scripts

步骤3:编写并运行助手脚本

Two runtime-generated helpers are used. Both follow the grep-and-regenerate pattern: before reuse, grep the first line for the version marker; if absent or a different integer, REGENERATE.
使用两个运行时生成的助手脚本。两者均遵循grep并重新生成的模式:复用前,grep第一行查找版本标记;若缺失或版本号不同,则重新生成。

Builder:
build_structural_index.py

构建器:
build_structural_index.py

  1. Write the builder to
    workspace/helpers/build_structural_index.py
    . The FIRST LINE MUST be exactly
    # MANTIS_HELPER_VERSION = 5
    . Before reusing an existing helper, grep its first lines for
    MANTIS_HELPER_VERSION = 5
    ; if that marker is absent or a different integer, REGENERATE the helper.
  2. The builder partitions the codebase into semantic units (see Per-Language Semantic Units), computes content-addressed cache keys, checks
    units/
    for reuse, selects a backend per partition (Step 2), extracts symbols + edges, and writes results to
    catalog.sqlite
    .
  3. The builder must extract:
    • Symbols:
      symbol_id
      ,
      name
      ,
      qualified_name
      ,
      namespace
      ,
      language
      ,
      file_path
      ,
      start_line
      ,
      end_line
      ,
      kind
      ,
      signature
      ,
      backend
      ,
      precision
      .
    • Call edges:
      caller_id
      ,
      callee_id
      ,
      callee_name
      ,
      file_path
      ,
      line
      ,
      edge_kind
      .
    • Function boundaries:
      symbol_id
      ,
      file_path
      ,
      start_line
      ,
      end_line
      ,
      signature
      ,
      language
      .
    • Coverage:
      file_path
      ,
      indexed
      ,
      backend
      ,
      precision
      ,
      unit_cache_key
      ,
      status
      .
  4. The builder writes the manifest LAST (Step 4) and the compatibility pointer.
  5. The builder MUST use ABSOLUTE paths and set its own working directory (Block A step 6). It MUST NOT write anything under
    CODE_ROOT
    when
    snapshot_pinned
    is true (Block A step 4). Backends that produce sidecar files (ctags
    tags
    , cscope
    cscope.out
    , clangd cache) MUST be redirected to STATE-RELATIVE paths:
    ctags -f <state>/helpers/tags
    ,
    cscope -f <state>/helpers/cscope.out
    ,
    CLANGD_INDEX_STORAGE=<state>/helpers/
    . Read-only LSP/SCIP queries to a running server need no redirect.
  1. 将构建器写入
    workspace/helpers/build_structural_index.py
    第一行必须精确为
    # MANTIS_HELPER_VERSION = 5
    。复用现有助手前,grep其前几行查找
    MANTIS_HELPER_VERSION = 5
    ;若标记缺失或版本号不同,则重新生成助手。
  2. 构建器将代码库划分为语义单元(见按语言划分的语义单元),计算基于内容寻址的缓存键,检查
    units/
    是否可复用,按分区选择后端(步骤2),提取符号+边,并将结果写入
    catalog.sqlite
  3. 构建器必须提取:
    • 符号
      symbol_id
      name
      qualified_name
      namespace
      language
      file_path
      start_line
      end_line
      kind
      signature
      backend
      precision
    • 调用边
      caller_id
      callee_id
      callee_name
      file_path
      line
      edge_kind
    • 函数边界
      symbol_id
      file_path
      start_line
      end_line
      signature
      language
    • 覆盖情况
      file_path
      indexed
      backend
      precision
      unit_cache_key
      status
  4. 构建器最后写入清单(步骤4)和兼容指针。
  5. 构建器必须使用绝对路径并设置自身工作目录(Block A步骤6)。当
    snapshot_pinned
    为true时,不得写入CODE_ROOT下的任何内容(Block A步骤4)。会生成附属文件的后端(如ctags的
    tags
    、cscope的
    cscope.out
    、clangd缓存)必须重定向到与状态相关的路径:
    ctags -f <state>/helpers/tags
    cscope -f <state>/helpers/cscope.out
    CLANGD_INDEX_STORAGE=<state>/helpers/
    。对运行中服务器的只读LSP/SCIP查询无需重定向。

Query helper:
query_structural_index.py

查询助手:
query_structural_index.py

  1. Write the query helper to
    workspace/helpers/query_structural_index.py
    . The FIRST LINE MUST be exactly
    # MANTIS_HELPER_VERSION = 5
    . Before reusing an existing helper, grep its first lines for
    MANTIS_HELPER_VERSION = 5
    ; if absent or a different integer, REGENERATE.
  2. The query helper provides bounded, paginated operations against
    catalog.sqlite
    (or a remote endpoint — identical API). It IS the consumption contract (see Query Interface below).
  3. If
    catalog.sqlite
    is absent but
    structural_index.jsonl
    exists, the query helper falls back to linear scanning of the JSONL file (slower but functional). If both are absent, it returns empty results with
    coverage.partition_status = "empty"
    .
  1. 将查询助手写入
    workspace/helpers/query_structural_index.py
    第一行必须精确为
    # MANTIS_HELPER_VERSION = 5
    。复用现有助手前,grep其前几行查找
    MANTIS_HELPER_VERSION = 5
    ;若缺失或版本号不同,则重新生成。
  2. 查询助手提供针对
    catalog.sqlite
    (或远程端点——API完全一致)的受限分页操作。它是消费约定的核心(见下文查询接口)。
  3. catalog.sqlite
    缺失但
    structural_index.jsonl
    存在,查询助手会回退到线性扫描JSONL文件(速度较慢但可用)。若两者均缺失,则返回空结果,且
    coverage.partition_status = "empty"

Step 4: Write Manifest and Commit

步骤4:写入清单并提交

  1. Write the manifest to
    workspace/kb/structural_index/tmp/manifest.json
    first. Then atomically rename it to
    workspace/kb/structural_index/manifest.json
    . This is the atomic commit point — the manifest is written LAST, after all units, catalog, and coverage data are written.
  2. Write a compatibility pointer to
    workspace/kb/structural_index.jsonl
    (STATE-RELATIVE — NEVER under
    CODE_ROOT
    ). Below a configurable threshold (default: 10K records), emit a complete JSONL export (provenance header + all records). Above the threshold, emit only a provenance header with
    compat_pointer.full_export = false
    and
    compat_pointer.symbol_count
    set. Large consumers MUST use the query interface.
  3. The manifest
    provider.backend_versions
    field records which backend was used per language (e.g.,
    {"go": {"backend_name": "scip-clangd", "precision": "semantic"}, "python": {"backend_name": "tree-sitter", "precision": "ast"}}
    ), so consumers know the precision level.
  4. Interrupted builds leave unreferenced temp objects in
    tmp/
    without corrupting the last published index. On resume, check
    manifest.json
    status and
    snapshot_id
    (Step 1).
  1. 首先将清单写入
    workspace/kb/structural_index/tmp/manifest.json
    。然后原子重命名为
    workspace/kb/structural_index/manifest.json
    。这是原子提交点——清单最后写入,在所有单元、目录和覆盖数据写入完成之后。
  2. 将兼容指针写入
    workspace/kb/structural_index.jsonl
    (与状态相关——绝不写入CODE_ROOT下)。低于可配置阈值(默认:10000条记录)时,导出完整的JSONL内容(溯源头 + 所有记录)。高于阈值时,仅写入包含
    compat_pointer.full_export = false
    compat_pointer.symbol_count
    的溯源头。大型消费者必须使用查询接口。
  3. 清单的
    provider.backend_versions
    字段记录每种语言使用的后端(例如
    {"go": {"backend_name": "scip-clangd", "precision": "semantic"}, "python": {"backend_name": "tree-sitter", "precision": "ast"}}
    ),以便消费者了解精度级别。
  4. 中断的构建会在
    tmp/
    中留下未引用的临时对象,但不会损坏最后发布的索引。恢复时,检查
    manifest.json
    的状态和
    snapshot_id
    (步骤1)。

Step 5: Return Results / Notify Caller

步骤5:返回结果/通知调用者

  1. Return the path to
    manifest.json
    ,
    catalog.sqlite
    , and the query helper. Include a summary (unit count, symbol count, call-edge count, backends used, coverage status).
  2. If the index is empty (no tools available or no source files found), notify the caller: "Structural index is empty — structural context unavailable."
  3. Do not notify the user directly — this skill is invoked as a sub-agent by the harness, planner, or researcher.
  1. 返回
    manifest.json
    catalog.sqlite
    和查询助手的路径。包含摘要(单元数量、符号数量、调用边数量、使用的后端、覆盖状态)。
  2. 若索引为空(无可用工具或未找到源代码文件),通知调用者:“结构索引为空——结构上下文不可用。”
  3. 请勿直接通知用户——该技能由harness、规划器或研究者作为子代理调用。

Index Schema and On-Disk Contract

索引架构与磁盘约定

On-Disk Layout

磁盘布局

workspace/kb/structural_index/
├── manifest.json          # Atomic commit point — written LAST
├── catalog.sqlite         # Query-optimized serving store (both directions indexed)
├── units/                 # Content-addressed immutable unit outputs
│   └── ab/cd/abcdef...    # sha256 prefix sharding (2+2 hex dirs)
├── shards/                # Partitioned serving data (large corpora)
│   └── shard_0000.sqlite
├── native/                # Prebuilt index attachments (SCIP, Kythe, LSIF)
│   ├── provenance.json    # Prebuilt index provenance manifest
│   ├── scip/
│   └── kythe/
└── tmp/                   # Temporary objects during build

workspace/helpers/
├── build_structural_index.py    # Builder (MANTIS_HELPER_VERSION = 5)
└── query_structural_index.py    # Query helper (MANTIS_HELPER_VERSION = 5)

workspace/kb/structural_index.jsonl  # Compatibility pointer
workspace/kb/structural_index/
├── manifest.json          # 原子提交点 —— 最后写入
├── catalog.sqlite         # 查询优化的服务存储(双向索引)
├── units/                 # 基于内容寻址的不可变单元输出
│   └── ab/cd/abcdef...    # sha256前缀分片(2+2十六进制目录)
├── shards/                # 分区服务数据(大型语料库)
│   └── shard_0000.sqlite
├── native/                # 预构建索引附件(SCIP、Kythe、LSIF)
│   ├── provenance.json    # 预构建索引溯源清单
│   ├── scip/
│   └── kythe/
└── tmp/                   # 构建过程中的临时对象

workspace/helpers/
├── build_structural_index.py    # 构建器(MANTIS_HELPER_VERSION = 5)
└── query_structural_index.py    # 查询助手(MANTIS_HELPER_VERSION = 5)

workspace/kb/structural_index.jsonl  # 兼容指针

Manifest Schema (
manifest.json
)

清单架构(
manifest.json

json
{
  "schema_version": 1,
  "snapshot_id": "<SNAPSHOT_ID or 'unknown'>",
  "root_fingerprint": "<sha256 of sorted (path, content_sha256) for all source files>",
  "status": "complete|partial|empty|failed",
  "provider": {
    "kind": "local-build|baseline+overlay|remote",
    "catalog": "catalog.sqlite",
    "backend_versions": {
      "cpp": {"backend_name": "tree-sitter", "backend_version": "0.20.8", "precision": "ast"},
      "go": {"backend_name": "scip-clangd", "backend_version": "0.2.3", "precision": "semantic"}
    }
  },
  "units": {"total": 0, "reused": 0, "rebuild": 0, "failed": 0},
  "coverage": {"total_files": 0, "indexed_files": 0, "failed_files": 0, "deferred_files": 0},
  "shards": [{"id": "", "path": "", "checksum": "", "partition_key": "", "symbol_count": 0, "edge_count": 0}],
  "deferred_units": [{"unit_id": "", "language": "", "files": [], "priority": 4, "reason": ""}],
  "native_indexes": [{"kind": "scip", "path": "", "snapshot_id": "", "root_fingerprint": "", "language": "", "indexer": "", "precision": ""}],
  "baseline": {"source": "ci|local|none", "snapshot_id": "", "manifest_path": ""},
  "overlay": {"units_added": 0, "units_modified": 0, "files": []},
  "compat_pointer": {"path": "structural_index.jsonl", "full_export": true, "symbol_count": 0},
  "created_at": "<ISO 8601>",
  "build_duration_ms": 0
}
Atomic commit: The manifest is written LAST (atomic rename from
tmp/
). Interrupted builds leave unreferenced temp objects without corrupting the last published index.
json
{
  "schema_version": 1,
  "snapshot_id": "<SNAPSHOT_ID or 'unknown'>",
  "root_fingerprint": "<sha256 of sorted (path, content_sha256) for all source files>",
  "status": "complete|partial|empty|failed",
  "provider": {
    "kind": "local-build|baseline+overlay|remote",
    "catalog": "catalog.sqlite",
    "backend_versions": {
      "cpp": {"backend_name": "tree-sitter", "backend_version": "0.20.8", "precision": "ast"},
      "go": {"backend_name": "scip-clangd", "backend_version": "0.2.3", "precision": "semantic"}
    }
  },
  "units": {"total": 0, "reused": 0, "rebuild": 0, "failed": 0},
  "coverage": {"total_files": 0, "indexed_files": 0, "failed_files": 0, "deferred_files": 0},
  "shards": [{"id": "", "path": "", "checksum": "", "partition_key": "", "symbol_count": 0, "edge_count": 0}],
  "deferred_units": [{"unit_id": "", "language": "", "files": [], "priority": 4, "reason": ""}],
  "native_indexes": [{"kind": "scip", "path": "", "snapshot_id": "", "root_fingerprint": "", "language": "", "indexer": "", "precision": ""}],
  "baseline": {"source": "ci|local|none", "snapshot_id": "", "manifest_path": ""},
  "overlay": {"units_added": 0, "units_modified": 0, "files": []},
  "compat_pointer": {"path": "structural_index.jsonl", "full_export": true, "symbol_count": 0},
  "created_at": "<ISO 8601>",
  "build_duration_ms": 0
}
原子提交: 清单最后写入(从
tmp/
原子重命名)。中断的构建会留下未引用的临时对象,但不会损坏最后发布的索引。

SQLite Catalog Schema (
catalog.sqlite
)

SQLite目录架构(
catalog.sqlite

sql
CREATE TABLE IF NOT EXISTS schema_meta (
    key TEXT PRIMARY KEY, value TEXT NOT NULL
);

CREATE TABLE IF NOT EXISTS symbols (
    symbol_id       TEXT PRIMARY KEY,
    name            TEXT NOT NULL,
    qualified_name  TEXT NOT NULL,
    namespace       TEXT,
    language        TEXT NOT NULL,
    file_path       TEXT NOT NULL,
    start_line      INTEGER NOT NULL,
    end_line        INTEGER,
    kind            TEXT NOT NULL,
    signature       TEXT,
    backend         TEXT NOT NULL,
    precision       TEXT NOT NULL CHECK (precision IN ('semantic','typecheck','ast','symbol-only','heuristic','deferred','coverage-only')),
    corpus          TEXT DEFAULT 'default',
    partition_key   TEXT,
    unit_cache_key  TEXT,
    source_layer    TEXT NOT NULL DEFAULT 'baseline'
);

CREATE TABLE IF NOT EXISTS call_edges (
    edge_id     INTEGER PRIMARY KEY AUTOINCREMENT,
    caller_id   TEXT NOT NULL,
    callee_id   TEXT,
    callee_name TEXT NOT NULL,
    file_path   TEXT NOT NULL,
    line        INTEGER NOT NULL,
    edge_kind   TEXT NOT NULL CHECK (edge_kind IN ('direct','indirect','virtual','macro','unresolved')),
    corpus      TEXT DEFAULT 'default',
    partition_key TEXT,
    source_layer TEXT NOT NULL DEFAULT 'baseline',
    FOREIGN KEY (caller_id) REFERENCES symbols(symbol_id),
    FOREIGN KEY (callee_id) REFERENCES symbols(symbol_id)
);

CREATE TABLE IF NOT EXISTS function_boundaries (
    symbol_id   TEXT PRIMARY KEY,
    file_path   TEXT NOT NULL,
    start_line  INTEGER NOT NULL,
    end_line    INTEGER NOT NULL,
    signature   TEXT,
    language    TEXT NOT NULL,
    FOREIGN KEY (symbol_id) REFERENCES symbols(symbol_id)
);

CREATE TABLE IF NOT EXISTS coverage (
    file_path   TEXT NOT NULL,
    indexed     INTEGER NOT NULL DEFAULT 0,
    backend     TEXT,
    precision   TEXT,
    unit_cache_key TEXT,
    status      TEXT NOT NULL DEFAULT 'pending',
    corpus      TEXT DEFAULT 'default',
    partition_key TEXT,
    PRIMARY KEY (file_path, corpus)
);

CREATE TABLE IF NOT EXISTS unit_cache (
    cache_key       TEXT PRIMARY KEY,
    unit_id         TEXT NOT NULL,
    language        TEXT NOT NULL,
    extractor_name  TEXT NOT NULL,
    extractor_version TEXT NOT NULL,
    file_count      INTEGER NOT NULL,
    symbol_count    INTEGER NOT NULL,
    edge_count      INTEGER NOT NULL,
    source_bytes    INTEGER NOT NULL,
    created_at      TEXT NOT NULL,
    snapshot_id     TEXT NOT NULL
);

CREATE TABLE IF NOT EXISTS deferred_units (
    unit_id     TEXT PRIMARY KEY,
    language    TEXT NOT NULL,
    files       TEXT NOT NULL,
    priority    INTEGER NOT NULL,
    reason      TEXT NOT NULL,
    cache_key   TEXT,
    created_at  TEXT NOT NULL
);

-- Indexes: both directions!
CREATE INDEX IF NOT EXISTS idx_symbols_name ON symbols(name);
CREATE INDEX IF NOT EXISTS idx_symbols_qualified ON symbols(qualified_name);
CREATE INDEX IF NOT EXISTS idx_symbols_file ON symbols(file_path, start_line);
CREATE INDEX IF NOT EXISTS idx_symbols_lang ON symbols(language);
CREATE INDEX IF NOT EXISTS idx_edges_caller ON call_edges(caller_id);
CREATE INDEX IF NOT EXISTS idx_edges_callee ON call_edges(callee_id);
CREATE INDEX IF NOT EXISTS idx_edges_callee_name ON call_edges(callee_name);
CREATE INDEX IF NOT EXISTS idx_edges_file ON call_edges(file_path, line);
CREATE INDEX IF NOT EXISTS idx_boundaries_file ON function_boundaries(file_path, start_line, end_line);
CREATE INDEX IF NOT EXISTS idx_coverage_status ON coverage(status);
CREATE INDEX IF NOT EXISTS idx_deferred_priority ON deferred_units(priority);
Partitioning: When symbols exceed 500K or edges exceed 2M, partition into shard SQLite files under
shards/
by
lang:{language}:bucket:{NN}
(stable hash). Each shard has the same schema. The query helper routes queries to the correct shard(s) based on the manifest.
sql
CREATE TABLE IF NOT EXISTS schema_meta (
    key TEXT PRIMARY KEY, value TEXT NOT NULL
);

CREATE TABLE IF NOT EXISTS symbols (
    symbol_id       TEXT PRIMARY KEY,
    name            TEXT NOT NULL,
    qualified_name  TEXT NOT NULL,
    namespace       TEXT,
    language        TEXT NOT NULL,
    file_path       TEXT NOT NULL,
    start_line      INTEGER NOT NULL,
    end_line        INTEGER,
    kind            TEXT NOT NULL,
    signature       TEXT,
    backend         TEXT NOT NULL,
    precision       TEXT NOT NULL CHECK (precision IN ('semantic','typecheck','ast','symbol-only','heuristic','deferred','coverage-only')),
    corpus          TEXT DEFAULT 'default',
    partition_key   TEXT,
    unit_cache_key  TEXT,
    source_layer    TEXT NOT NULL DEFAULT 'baseline'
);

CREATE TABLE IF NOT EXISTS call_edges (
    edge_id     INTEGER PRIMARY KEY AUTOINCREMENT,
    caller_id   TEXT NOT NULL,
    callee_id   TEXT,
    callee_name TEXT NOT NULL,
    file_path   TEXT NOT NULL,
    line        INTEGER NOT NULL,
    edge_kind   TEXT NOT NULL CHECK (edge_kind IN ('direct','indirect','virtual','macro','unresolved')),
    corpus      TEXT DEFAULT 'default',
    partition_key TEXT,
    source_layer TEXT NOT NULL DEFAULT 'baseline',
    FOREIGN KEY (caller_id) REFERENCES symbols(symbol_id),
    FOREIGN KEY (callee_id) REFERENCES symbols(symbol_id)
);

CREATE TABLE IF NOT EXISTS function_boundaries (
    symbol_id   TEXT PRIMARY KEY,
    file_path   TEXT NOT NULL,
    start_line  INTEGER NOT NULL,
    end_line    INTEGER NOT NULL,
    signature   TEXT,
    language    TEXT NOT NULL,
    FOREIGN KEY (symbol_id) REFERENCES symbols(symbol_id)
);

CREATE TABLE IF NOT EXISTS coverage (
    file_path   TEXT NOT NULL,
    indexed     INTEGER NOT NULL DEFAULT 0,
    backend     TEXT,
    precision   TEXT,
    unit_cache_key TEXT,
    status      TEXT NOT NULL DEFAULT 'pending',
    corpus      TEXT DEFAULT 'default',
    partition_key TEXT,
    PRIMARY KEY (file_path, corpus)
);

CREATE TABLE IF NOT EXISTS unit_cache (
    cache_key       TEXT PRIMARY KEY,
    unit_id         TEXT NOT NULL,
    language        TEXT NOT NULL,
    extractor_name  TEXT NOT NULL,
    extractor_version TEXT NOT NULL,
    file_count      INTEGER NOT NULL,
    symbol_count    INTEGER NOT NULL,
    edge_count      INTEGER NOT NULL,
    source_bytes    INTEGER NOT NULL,
    created_at      TEXT NOT NULL,
    snapshot_id     TEXT NOT NULL
);

CREATE TABLE IF NOT EXISTS deferred_units (
    unit_id     TEXT PRIMARY KEY,
    language    TEXT NOT NULL,
    files       TEXT NOT NULL,
    priority    INTEGER NOT NULL,
    reason      TEXT NOT NULL,
    cache_key   TEXT,
    created_at  TEXT NOT NULL
);

-- 索引:双向!
CREATE INDEX IF NOT EXISTS idx_symbols_name ON symbols(name);
CREATE INDEX IF NOT EXISTS idx_symbols_qualified ON symbols(qualified_name);
CREATE INDEX IF NOT EXISTS idx_symbols_file ON symbols(file_path, start_line);
CREATE INDEX IF NOT EXISTS idx_symbols_lang ON symbols(language);
CREATE INDEX IF NOT EXISTS idx_edges_caller ON call_edges(caller_id);
CREATE INDEX IF NOT EXISTS idx_edges_callee ON call_edges(callee_id);
CREATE INDEX IF NOT EXISTS idx_edges_callee_name ON call_edges(callee_name);
CREATE INDEX IF NOT EXISTS idx_edges_file ON call_edges(file_path, line);
CREATE INDEX IF NOT EXISTS idx_boundaries_file ON function_boundaries(file_path, start_line, end_line);
CREATE INDEX IF NOT EXISTS idx_coverage_status ON coverage(status);
CREATE INDEX IF NOT EXISTS idx_deferred_priority ON deferred_units(priority);
分区: 当符号超过500K或边超过2M时,按
lang:{language}:bucket:{NN}
(稳定哈希)将数据分区到
shards/
下的分片SQLite文件中。每个分片具有相同的架构。查询助手根据清单将查询路由到正确的分片。

Content-Addressed Cache Key

基于内容寻址的缓存键

python
cache_key = sha256(
    schema_version +
    extractor_name@version +
    language +
    compile_context_digest +
    ordered_input_content_digests +
    dependency_or_interface_digest
)
Critical:
snapshot_id
is NOT in the cache key. It goes in provenance only. This permits reuse across commits, branches, and snapshots with identical units.
python
cache_key = sha256(
    schema_version +
    extractor_name@version +
    language +
    compile_context_digest +
    ordered_input_content_digests +
    dependency_or_interface_digest
)
关键:
snapshot_id
不在缓存键中。仅在溯源信息中记录。这允许跨提交、分支和具有相同单元的快照复用缓存。

Canonical Symbol IDs

规范符号ID

Native IDs (when semantic backend is available):
  • SCIP:
    scip:{symbol}
  • Kythe:
    kythe:{uri}
  • clangd:
    clangd:{usr}
Fallback IDs (when no semantic backend):
fallback:{language}:{file_path}:{sha256(qualified_name|start_line|signature)[:16]}
This distinguishes namespaces, overloads, methods, local functions, and duplicate names across languages or repositories.
原生ID(当语义后端可用时):
  • SCIP:
    scip:{symbol}
  • Kythe:
    kythe:{uri}
  • clangd:
    clangd:{usr}
回退ID(当无语义后端时):
fallback:{language}:{file_path}:{sha256(qualified_name|start_line|signature)[:16]}
这可区分不同语言或仓库中的命名空间、重载、方法、本地函数和重复名称。

Edge Kinds

调用边类型

Edge kinds are kept explicitly separate — never silently merged:
  • direct
    — statically resolved call to a known symbol
  • indirect
    — function pointer, closure, callback
  • virtual
    — virtual method dispatch (runtime-resolved)
  • macro
    — call introduced by macro expansion
  • unresolved
    — callee name found but no symbol_id resolved
调用边类型明确区分——绝不静默合并:
  • direct
    —— 静态解析到已知符号的调用
  • indirect
    —— 函数指针、闭包、回调
  • virtual
    —— 虚方法调度(运行时解析)
  • macro
    —— 宏展开引入的调用
  • unresolved
    —— 找到被调用者名称但未解析到symbol_id

Per-Language Semantic Units

按语言划分的语义单元

LanguageUnit TypeCompile ContextDependency Digest
C/C++Compilation unitcompile_commands.json entrysha256 of transitive header interfaces
GoPackagego.mod + build tagssha256 of imported packages' exported API
RustCrateCargo.toml + featuressha256 of extern crate signatures
Java/KotlinCompiler batchmodule path + classpathsha256 of imported class signatures
TypeScriptProjecttsconfig.jsonsha256 of imported module type declarations
FallbackIndividual fileNoneEmpty string
Fallback Granularity Rule: For the fallback tier (regex, AST parser, or lightweight crawler without a language build system), each source file MUST be its own independent semantic unit. Never bundle multiple source files into a single fallback unit. Per-file unit isolation is essential for content-addressed cache efficiency — when one file changes in a future snapshot, only that single file's unit invalidates while all unchanged files hit the cache (
reused = N-1
).
语言单元类型编译上下文依赖摘要
C/C++编译单元compile_commands.json条目传递性头文件接口的sha256值
Gogo.mod + 构建标签导入包导出API的sha256值
RustCrateCargo.toml + 特性外部crate签名的sha256值
Java/Kotlin编译器批次模块路径 + 类路径导入类签名的sha256值
TypeScript项目tsconfig.json导入模块类型声明的sha256值
回退单个文件空字符串
回退粒度规则: 对于回退层级(正则、AST解析器或无语言构建系统的轻量级爬虫),每个源代码文件必须作为独立的语义单元。绝不能将多个源代码文件捆绑到单个回退单元中。按文件划分单元的隔离性对于基于内容寻址的缓存效率至关重要——当未来快照中某个文件更改时,仅该文件的单元失效,所有未更改的文件均可命中缓存(
reused = N-1
)。

Baseline + Delta Overlay

基线+增量覆盖

  • Static baseline: Built once in CI for the complete snapshot.
  • Incremental cache: Immutable unit outputs reused across snapshots.
  • Delta overlay: Changed/target files indexed locally, layered over baseline.
  • Remote serving: Optional when global index is too costly locally.
At query time, the query helper merges baseline + overlay results. Overlay symbols supersede baseline symbols for the same file. The manifest records
provider.kind = "baseline+overlay"
with baseline manifest reference and overlay unit count.
  • 静态基线: 在CI中为完整快照构建一次。
  • 增量缓存: 不可变单元输出可跨快照复用。
  • 增量覆盖: 本地索引已更改/目标文件,叠加在基线上。
  • 远程服务: 当全局索引本地成本过高时可选。
查询时,查询助手会合并基线+覆盖结果。覆盖符号会替代基线中同一文件的符号。清单记录
provider.kind = "baseline+overlay"
,包含基线清单引用和覆盖单元数量。

Compatibility JSONL Export

兼容JSONL导出

The old
structural_index.jsonl
format is retained as a compatibility export only — it is NOT the query contract. Below the threshold (default 10K records), a complete JSONL export is emitted. Above the threshold, only a provenance pointer is written. Large consumers MUST use the query interface.
Line 1 — provenance header:
json
{"_provenance": true, "snapshot_id": "abc123", "tool": "tree-sitter"}
Lines 2+ — structural records (one per line,
_type
discriminator):
json
{"_type": "function", "key": "src/parser.c:parse_input", "start_line": 45, "end_line": 120, "signature": "int parse_input(char *buf, size_t len)", "calls": ["malloc", "validate_input", "memcpy"]}
{"_type": "call_edge", "caller": "parse_input", "callee": "malloc", "file": "src/parser.c", "line": 78}
{"_type": "call_edge", "caller": "main", "callee": "parse_input", "file": "src/main.c", "line": 203}
Provenance header fields:
FieldTypeDescription
_provenance
boolAlways
true
— marks this as the provenance header line
snapshot_id
string
SNAPSHOT_ID
the index was built against
tool
stringBackend used (e.g.
"lsp-clangd"
,
"tree-sitter"
,
"ctags"
,
"regex"
,
"grep"
)
Record types:
_type
DescriptionKey fields
function
Function definition with boundary and signature
key
,
start_line
,
end_line
,
signature
,
calls
call_edge
A call from caller to callee at file:line
caller
,
callee
,
file
,
line
旧的
structural_index.jsonl
格式仅作为兼容导出保留——它不是查询约定。低于阈值(默认10000条记录)时,导出完整的JSONL内容。高于阈值时,仅写入溯源指针。大型消费者必须使用查询接口。
第1行——溯源头:
json
{"_provenance": true, "snapshot_id": "abc123", "tool": "tree-sitter"}
第2行及以后——结构记录(每行一条,
_type
作为区分符):
json
{"_type": "function", "key": "src/parser.c:parse_input", "start_line": 45, "end_line": 120, "signature": "int parse_input(char *buf, size_t len)", "calls": ["malloc", "validate_input", "memcpy"]}
{"_type": "call_edge", "caller": "parse_input", "callee": "malloc", "file": "src/parser.c", "line": 78}
{"_type": "call_edge", "caller": "main", "callee": "parse_input", "file": "src/main.c", "line": 203}
溯源头字段:
字段类型描述
_provenance
bool始终为
true
——标记此行是溯源头
snapshot_id
string构建索引所针对的
SNAPSHOT_ID
tool
string使用的后端(例如
"lsp-clangd"
"tree-sitter"
"ctags"
"regex"
"grep"
记录类型:
_type
描述关键字段
function
包含边界和签名的函数定义
key
start_line
end_line
signature
calls
call_edge
文件某一行中调用者到被调用者的调用关系
caller
callee
file
line

Snapshot Safety

快照安全性

  1. Build from
    CODE_ROOT
    , not live tree (when pinned).
    When the snapshot is pinned, the structural index is built from the pinned
    CODE_ROOT
    , ensuring it reflects the exact bytes the pipeline is analyzing.
  2. Reuse-on-match. If
    manifest.json
    already carries the current
    SNAPSHOT_ID
    (and
    SNAPSHOT_ID
    is NOT
    "unknown"
    — in MODE-OFF, always rebuild), reuse it — do not rebuild. Rebuild when
    SNAPSHOT_ID
    differs or is
    "unknown"
    . Individual units may still be reused from the content-addressed cache. This bounds cost across retries/crash-resume.
  3. Manifest atomicity. The manifest is written LAST via atomic rename from
    tmp/
    . Interrupted builds leave unreferenced temp objects without corrupting the last published index. On resume, check
    manifest.json
    status and
    snapshot_id
    .
  4. STALE flag in HALT mode. When
    snapshot_pinned
    is false (HALT), the index may be built from the unpinned
    CODE_ROOT
    but is marked as potentially stale. Consumers treat structural hints as advisory.
  5. MODE-OFF: build, do not skip. When
    active_snapshot
    is absent (MODE-OFF), build against the current directory (cwd) with provenance
    snapshot_id
    set to
    "unknown"
    . This is the standalone-efficiency case — the index is still useful for the researcher/planner even without snapshot pinning. Do NOT return an empty index merely because the snapshot is absent.
  1. CODE_ROOT
    构建,而非实时代码树(当快照固定时)。
    当快照固定时,结构索引从固定的
    CODE_ROOT
    构建,确保其反映流水线正在分析的精确字节。
  2. 匹配时复用。
    manifest.json
    已包含当前
    SNAPSHOT_ID
    (且
    SNAPSHOT_ID
    不为
    "unknown"
    ——MODE-OFF模式下始终重建),则复用该索引——不重建。当
    SNAPSHOT_ID
    不同或为
    "unknown"
    时重建。单个单元仍可从基于内容寻址的缓存中复用。这可限制重试/崩溃恢复场景下的成本。
  3. 清单原子性。 清单最后通过从
    tmp/
    原子重命名写入。中断的构建会留下未引用的临时对象,但不会损坏最后发布的索引。恢复时,检查
    manifest.json
    的状态和
    snapshot_id
  4. HALT模式下的STALE标记。
    snapshot_pinned
    为false(HALT)时,索引可能从未固定的
    CODE_ROOT
    构建,但会标记为可能过期。消费者将结构提示视为建议性内容。
  5. MODE-OFF:构建,不跳过。
    active_snapshot
    缺失(MODE-OFF)时,针对当前目录(cwd)构建,溯源信息中的
    snapshot_id
    设为
    "unknown"
    。这是独立高效的使用场景——即使没有快照固定,索引对研究者/规划器仍然有用。请勿仅因快照缺失而返回空索引。

Consumption Contract

消费约定

These are runtime instructions for callers — they define how consumers use the structural index. The structural index is a HINT-only enhancement; skills that do not use it behave exactly as they do today.
这些是针对调用者的运行时指令——定义了消费者如何使用结构索引。结构索引仅作为提示性增强;不使用它的技能行为与当前完全一致。

Query Interface

查询接口

The query helper IS the contract. All consumers use
query_structural_index.py
(or a compatible remote endpoint). The JSONL file is NOT the query contract — it is a compatibility export only.
Operations:
  1. resolve_symbol(name, language?, file?, namespace?)
    {results, total, ambiguous, coverage}
    • Returns ALL matches — no silent selection among ambiguous symbols.
    • Caller MUST disambiguate before calling
      find_callers
      /
      find_callees
      .
  2. find_callers(symbol_id, limit=100, offset=0)
    {results, total, has_more, coverage}
    • Bounded, paginated caller lookup.
    • Each result carries
      precision
      ,
      backend
      ,
      edge_kind
      .
    • Empty results carry
      coverage.partition_status
      .
  3. find_callees(symbol_id, limit=100, offset=0)
    {results, total, has_more, coverage}
    • Same shape as
      find_callers
      .
  4. get_function_boundary(file, line)
    {symbol_id, start_line, end_line, signature, precision, backend}
  5. get_coverage(file?)
    {total_files, indexed_files, failed, deferred, partition_status, backends_used}
Key properties:
  • Bounded results and pagination (prevents loading entire index into memory).
  • Explicit name resolution before graph traversal (no silent selection among ambiguous symbols).
  • Precision and backend attached to every result.
  • Coverage attached to empty results.
  • Identical operations for local and remote providers (remote via
    MANTIS_STRUCTURAL_INDEX_URL
    env var or manifest
    provider.kind = "remote"
    ).
Coverage on empty results:
partition_status
MeaningConsumer action
complete
All files indexed"No indexed callers" (still run grep per HINT-only rule)
partial
Some files deferred or failed"Not fully indexed" — MUST run grep fallback
empty
No backend available"Not indexed" — MUST run grep fallback
failed
Backend attempted but failed"Index failed" — MUST run grep fallback
查询助手是约定的核心。 所有消费者使用
query_structural_index.py
(或兼容的远程端点)。JSONL文件不是查询约定——它仅作为兼容导出。
操作:
  1. resolve_symbol(name, language?, file?, namespace?)
    {results, total, ambiguous, coverage}
    • 返回所有匹配结果——绝不静默选择歧义符号。
    • 调用者在调用
      find_callers
      /
      find_callees
      之前必须消除歧义。
  2. find_callers(symbol_id, limit=100, offset=0)
    {results, total, has_more, coverage}
    • 受限的分页调用者查找。
    • 每个结果包含
      precision
      backend
      edge_kind
    • 空结果包含
      coverage.partition_status
  3. find_callees(symbol_id, limit=100, offset=0)
    {results, total, has_more, coverage}
    • find_callers
      格式相同。
  4. get_function_boundary(file, line)
    {symbol_id, start_line, end_line, signature, precision, backend}
  5. get_coverage(file?)
    {total_files, indexed_files, failed, deferred, partition_status, backends_used}
关键特性:
  • 受限结果和分页(防止将整个索引加载到内存中)。
  • 图遍历前明确的名称解析(绝不静默选择歧义符号)。
  • 每个结果附带精度和后端信息。
  • 空结果附带覆盖情况。
  • 本地和远程提供者的操作完全一致(远程通过
    MANTIS_STRUCTURAL_INDEX_URL
    环境变量或清单
    provider.kind = "remote"
    实现)。
空结果的覆盖情况:
partition_status
含义消费者操作
complete
所有文件已索引“无索引调用者”(仍需按仅提示规则运行grep)
partial
部分文件延迟或失败“未完全索引”——必须运行grep回退
empty
无可用后端“未索引”——必须运行grep回退
failed
后端尝试但失败“索引失败”——必须运行grep回退

mantis-plan

mantis-plan

  • Use the structural index query helper for function-level dependency fan-out. When planning investigations, call
    resolve_symbol()
    then
    find_callers()
    to identify all functions that call into a target — this broadens the audit set beyond single-file analysis.
  • The structural index decides ORDER of investigations (which functions to audit first based on call-graph centrality), never MEMBERSHIP. It may broaden the audit set (safe over-reporting), but must never REMOVE or drop a file; the planner's existing logic remains the membership floor.
  • 使用结构索引查询助手进行函数级依赖扩散。规划调查时,调用
    resolve_symbol()
    然后
    find_callers()
    以识别所有调用目标函数的函数——这将审计范围扩展到单文件分析之外。
  • 结构索引决定调查顺序(基于调用图中心性优先审计哪些函数),但绝不决定审计范围。它可能扩大审计范围(安全的过度报告),但绝不能移除或丢弃任何文件;规划器的现有逻辑仍是审计范围的下限。

mantis-researcher

mantis-researcher

  • Wave 1 (Rapid Triage): Run a repo-wide grep for the function name to build the exhaustive set of candidate call-sites — this is the mandatory floor. Then use the structural index query helper (
    resolve_symbol
    then
    find_callers
    ) to RANK and prioritize which call-sites to audit first (the index distinguishes actual calls from comments/strings/variable names). Audit the union of both result sets — the structural index may miss macro-based calls, function pointers, and dynamic dispatch, so grep remains the floor.
  • Wave 2 (Deep Audit): Use
    get_function_boundary(file, line)
    to start with the enclosing function, expanding to callers/callees/file as needed for cross-function context — this saves context while preserving coverage.
  • If the structural index is absent or empty, fall back to grep-based discovery (today's behavior). The structural index is a coverage HINT only — it improves audit quality but is never required.
  • 第一阶段(快速分类): 对函数名称进行仓库级grep以构建完整的候选调用站点集合——这是强制下限。然后使用结构索引查询助手(
    resolve_symbol
    然后
    find_callers
    )对调用站点进行排序和优先级划分(索引可区分实际调用与注释/字符串/变量名)。审计两个结果集的并集——索引可能遗漏基于宏的调用、函数指针和动态调度,因此grep仍是下限。
  • 第二阶段(深度审计): 使用
    get_function_boundary(file, line)
    从包含的函数开始,根据需要扩展到调用者/被调用者/文件以获取跨函数上下文——这在保留覆盖范围的同时节省上下文。
  • 若结构索引缺失或为空,回退到基于grep的发现(当前行为)。结构索引仅作为覆盖提示——它提升审计质量但绝非必需。

Safety

安全性

Non-negotiable invariants:
  1. Agnostic. Nothing is ever required; no-tool / parse-fail / not-invoked → empty index → grep fallback = today's behavior byte-for-byte. Optional in the Pass Lifecycle Contract; a non-conformant harness simply skips it.
  2. HINT-only / union / never MEMBERSHIP. Consumers audit the union with grep; a symbol the index misses must still be reachable by the exhaustive sweep. The structural index decides ORDER, never MEMBERSHIP. The query contract enforces this:
    find_callers
    returns HINTs, never authoritative membership — consumers MUST union with grep.
  3. No verdicts, touches no findings. Cannot violate INV-1 and cannot itself drop a finding.
  4. Narrow scope: source cross-reference only. Binary/build-derived reachability ("is it compiled into production") is explicitly out of scope — a dev customization, not part of this skill, because absence-from-a-build can hide a real finding (INV-2). Do not fold build-derived reachability into this skill.
A reference blueprint is available at mantis-pipeline-adapter/references/mantis-structural-index.md. It is a stub that points to this SKILL.md as the single source of truth — do not duplicate spec content there.
不可协商的不变量:
  1. 无强制要求。 绝不强制要求任何内容;无工具/解析失败/未调用→空索引→grep回退=与当前行为完全一致。在Pass Lifecycle Contract中是可选的;不符合要求的harness可直接跳过它。
  2. 仅提示/并集/绝不决定范围。 消费者审计与grep结果的并集;索引遗漏的符号仍必须可通过 exhaustive sweep 访问。结构索引决定顺序,绝不决定范围。查询约定强制执行此规则:
    find_callers
    返回提示,绝非权威范围——消费者必须与grep结果取并集。
  3. 无结论,不触及发现结果。 不能违反INV-1,也不能自行丢弃任何发现结果。
  4. 范围狭窄:仅源代码交叉引用。 二进制/构建派生的可达性(“是否编译到生产环境中”)明确超出范围——这是开发者自定义内容,不属于该技能,因为构建中缺失可能隐藏真实发现结果(INV-2)。请勿将构建派生的可达性纳入该技能。
参考蓝图可在mantis-pipeline-adapter/references/mantis-structural-index.md获取。它是一个存根,指向本SKILL.md作为唯一的事实来源——请勿在那里复制规范内容。