Back to Details

plugin-audit

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

Plugin Audit

插件审计

Audit how well vercel-plugin skill injection performs on real-world Claude Code sessions.
审计vercel-plugin的skill injection功能在真实场景的Claude Code会话中的表现。

Workflow

工作流

1. Locate conversation logs

1. 定位对话日志

Find JSONL conversation logs for a target project:
bash
ls -lt ~/.claude/projects/-Users-*-<project-name>/*.jsonl
The path uses the project's absolute path with slashes replaced by hyphens and a leading hyphen.
找到目标项目的JSONL对话日志：
bash
ls -lt ~/.claude/projects/-Users-*-<project-name>/*.jsonl
路径使用项目的绝对路径，斜杠被替换为连字符且开头带一个连字符。

2. Extract tool calls

2. 提取工具调用

Parse the JSONL log to extract all tool_use entries. Each line is a JSON object with 
message.content[]
containing 
type: "tool_use"
blocks. Extract 
name
and 
input
fields. Group by tool type (Bash, Read, Write, Edit).
解析JSONL日志以提取所有tool_use条目。每一行都是一个JSON对象，其
message.content[]
中包含
type: "tool_use"
块。提取
name
和
input
字段，按工具类型（Bash、Read、Write、Edit）分组。

3. Test hook matching

3. 测试钩子匹配

Use the exported pipeline functions directly — do NOT shell out to the hook script for each test. Import from the hooks directory:
js
import { loadSkills, matchSkills } from "./hooks/pretooluse-skill-inject.mjs";
import { createLogger } from "./hooks/logger.mjs";
Call 
loadSkills()
once, then 
matchSkills(toolName, toolInput, compiledSkills)
for each tool call. This is fast and gives exact match results.
直接使用导出的流水线函数——请勿为每个测试单独执行钩子脚本。从hooks目录导入：
js
import { loadSkills, matchSkills } from "./hooks/pretooluse-skill-inject.mjs";
import { createLogger } from "./hooks/logger.mjs";
调用一次
loadSkills()
，然后为每个工具调用执行
matchSkills(toolName, toolInput, compiledSkills)
。这个方法速度很快，且能返回精确的匹配结果。

4. Identify gaps

4. 识别覆盖缺口

Compare matched skills against what SHOULD have matched based on the project's technology stack. Common gap categories:
  • Path pattern gaps: Files that should trigger a skill but don't (e.g., 
    src/db/schema.ts
    not matching 
    vercel-storage
    )
  • Bash pattern gaps: Commands that should trigger but don't (e.g., missing package manager variants)
  • Dedup masking: Skills that matched but were deduped before injection
  • Budget/cap drops: Skills matched but dropped by the 12KB budget or 3-skill ceiling
将匹配到的skill与基于项目技术栈预期应该匹配的skill做对比。常见的缺口类别：
  • 路径模式缺口：本应触发skill的文件没有触发（例如
    src/db/schema.ts
    没有匹配到
    vercel-storage
    ）
  • Bash模式缺口：本应触发的命令没有触发（例如缺失了包管理器的不同变体匹配）
  • 去重屏蔽：已经匹配到的skill在注入前被去重逻辑移除
  • 预算/容量限制丢弃：skill成功匹配，但因为12KB大小预算或者最多3个skill的上限被丢弃

5. Check plugin cache staleness

5. 检查插件缓存过期

Compare the installed plugin cache against the dev version:
bash
undefined
将已安装的插件缓存和开发版本做对比：
bash
undefined

Cache location

Cache location

~/.claude/plugins/cache/vercel-labs-vercel-plugin/vercel-plugin/<version>/
~/.claude/plugins/cache/vercel-labs-vercel-plugin/vercel-plugin/<version>/

Compare skill content

Compare skill content

diff <(grep 'pattern' skills/<skill>/SKILL.md) <(grep 'pattern' ~/.claude/plugins/cache/.../skills/<skill>/SKILL.md)

Check `~/.claude/plugins/installed_plugins.json` for version and git SHA.
diff <(grep 'pattern' skills/<skill>/SKILL.md) <(grep 'pattern' ~/.claude/plugins/cache/.../skills/<skill>/SKILL.md)

查看`~/.claude/plugins/installed_plugins.json`确认版本和git SHA值。

Report Format

报告格式

Produce a structured report with:
  1. Session summary: Project, date, tool call count, model
  2. Match matrix: Table of tool calls × matched skills (with match type)
  3. Coverage gaps: Unmatched tool calls that should have matched, with suggested pattern additions
  4. Dedup timeline: Order of skill injections and what got deduped
  5. Cache status: Whether installed version matches dev, with specific diffs
输出结构化报告，包含以下内容：
  1. 会话概览：项目、日期、工具调用数量、使用的模型
  2. 匹配矩阵：工具调用 × 匹配到的skill的表格（标注匹配类型）
  3. 覆盖缺口：本应匹配但未匹配的工具调用，附带建议新增的匹配模式
  4. 去重时间线：skill注入的顺序以及被去重的条目
  5. 缓存状态：已安装版本是否和开发版本一致，附带具体的差异内容

References

参考资料

  • Log format details
  • Test script for batch matching
  • 日志格式详情
  • 批量匹配测试脚本