read_knowledge impact audit
Create a report on the impact of the read_knowledge Dosu MCP on my agent trajectories.
Please analyze my historic agent sessions over the past 1-month and identify both highlights and failures of the read_knowledge MCP.
A highlight is when the information from the read_knowledge call was relevant to the question and solution. The agent does not have to uniquely cite it in plan or code.
A failure is when the agent found the information distracting or misleading.
For each highlight and failure, format it as
- task -> what the agent was working on
- knowledge -> what information was surfaced
- impact -> what impact it had
Write an HTML report with the results.
Product
- Inventory local agent logs (default: past 30 days, all projects). Cursor / Claude / Codex adapters plus a generic JSON/JSONL walker (Devin, Continue, Windsurf, , …).
- Extract every call and its tool result
- Judge each call: was the information relevant to the question and solution (not “did the agent uniquely cite it”)
- Open the HTML report (
generate_impact_report.py --open
Do
not call
. This skill only reports.
Session viewer
Every highlight must include a Review session control that opens an
inline transcript viewer inside the report.
The viewer must:
- Show a chronological, bounded window around the highlighted
call.
- Visually pin and distinguish:
- the user's task,
- the query sent to Dosu,
- the knowledge Dosu returned,
- the agent's subsequent reasoning, actions, and answer.
- Show the complete sanitized Dosu result when the source log preserved it.
- Clearly say "Result payload unavailable in this log source" when it cannot
be recovered; never imply the 400-character preview is complete.
- Show the agent's other tool calls in the window as compact action turns
with sanitized, truncated input and output previews — a bare tool name
gives no context for judging the call. Include the agent's reasoning
(thinking) turns before and after the call when the log records them.
- Use a self-contained HTML dialog or drawer with no network dependency.
- Support keyboard navigation, Escape-to-close, readable code blocks, and
copy buttons for the query and knowledge result.
- Exclude system/developer instructions, secrets, PII, and receipt IDs;
never include full raw payloads of other tools — bounded previews only.
- Identify the source and session, but do not expose raw filesystem paths.
The summary card remains concise. Transcript detail belongs only in the
session viewer.
Fresh run: always re-extract and re-classify. Ignore
,
, and any existing HTML. Do not skip because a previous report exists.
Do not ask
Never ask which agent, date range, project, or granularity. Defaults:
| Decision | Default |
|---|
| Window | |
| Projects | (every workspace on this machine) |
| Sources | cursor, claude, codex, generic |
Override only when the user already said so (“this repo”, “last week”, “Claude only”).
- Classification: references/classification.md
Workflow
Progress:
- [ ] 0. SKILL_DIR (this SKILL.md’s folder)
- [ ] 1. Extract read_knowledge calls (overwrite /tmp)
- [ ] 2. Classify every call from scratch
- [ ] 3. Write findings JSON
- [ ] 4. Open HTML report → short reply
Step 0 — Skill dir
is the directory that contains
this (the file you are reading now). Confirm the new classifier is loaded:
bash
test -f "$SKILL_DIR/scripts/extract_read_knowledge.py"
grep -F '"relevant": "Returned information was relevant to the question and solution."' \
"$SKILL_DIR/scripts/generate_impact_report.py"
grep -F "details class='fold'" "$SKILL_DIR/scripts/generate_impact_report.py"
grep -F "Result payload unavailable in this log source" \
"$SKILL_DIR/scripts/generate_impact_report.py"
If any grep fails, stop — you have a stale copy. Use the
checkout at
skills/read-knowledge-impact/
.
Then clear previous artifacts:
bash
rm -f /tmp/rk-calls.json /tmp/rk-findings.json /tmp/read-knowledge-impact.html
Step 1 — Extract
bash
# Set DAYS from the user's window BEFORE extract. The HTML reads this number.
# past day / last 24 hours / today → DAYS=1
# last week → DAYS=7
# last N days → DAYS=N
# unspecified → DAYS=30
DAYS=30
python3 "$SKILL_DIR/scripts/extract_read_knowledge.py" \
--days "$DAYS" --all-projects \
--out /tmp/rk-calls.json
| User says | Flags |
|---|
| (default) | |
| "this project" / "this repo" | (drop ) |
| "last N days" / "past day" / "last 24 hours" | ( for a day) |
| "Claude only" | --sources claude --days 30 --all-projects
|
filters by
call time (
from Cursor
/ Claude
), not file mtime. A long chat last-touched today does not count last week's calls.
Each call has
,
,
,
,
,
,
(
/
/
/
/
), and the session’s first user task — plus its stable location (
,
/
) and a
: a sanitized, bounded transcript window around the call with the complete Dosu result when the source log preserved it (
). Cursor JSONL transcripts generally omit tool-result payloads — the viewer then shows the query and downstream context with an honest unavailable-result state; oversized results are recovered from
sidecars when possible.
If the extractor prints
, open an empty report anyway and stop.
Step 2 — Classify
Read references/classification.md. Classify from the transcripts, not from a previous findings file.
- Keep mechanical hints (, , , ) unless the transcript clearly contradicts them.
- For , digest the session around that call — start from the call's , then
parse_agent_logs.py --digest <id>
- Set to exactly one of: , , , , , , .
- Fill , , for every and call, and for overflow/error when you can see what happened. Complete sentences — never cut a field mid-word. The report folds long copy behind “more”. No raw prompts, no secrets.
= the returned information was
relevant to the question and solution. The agent does not have to uniquely cite it. If they also grepped or read code, still mark
.
= a result came back but it was not about this question or solution.
= the result sent the agent the wrong way, contradicted the codebase, or crowded out the real answer.
Never use
. On-topic returns that were not uniquely quoted are
.
Step 3 — Findings file
Write
as
{ "window": …, "calls": [ … ] }
. Copy
from
unchanged (that is how the HTML knows it was 1 day vs 30). Each call is the extractor row
plus /
/
/
. Keep extractor fields — in particular carry
through unchanged (you may drop a turn that leaked something sensitive, never add or rewrite turns). Every
call must keep its
; the report generator refuses to build a highlight without a working viewer. Classify
every call — do not sample.
Do
not set
. The Outcomes table copy comes from
generate_impact_report.py
. The relevant row must read exactly: "Returned information was relevant to the question and solution."
Step 4 — Report
bash
python3 "$SKILL_DIR/scripts/generate_impact_report.py" \
--findings /tmp/rk-findings.json \
--out /tmp/read-knowledge-impact.html --open
The generator asserts that every highlight has a working
Review session viewer (a
with turns) and exits with an error otherwise — fix the findings, do not strip the field.
Off-topic, empty, and rejected calls are not hidden: they render in a collapsed No effect section at the bottom, each with its own Review session viewer, so a no-effect label can be audited the same way a highlight can.
Reply with the headline numbers (calls, % relevant, highlight count, failure count) and that the HTML is open. Do not paste every card into chat.
Guardrails
- Never write secrets / PII / raw log dumps into the report.
- Never invent calls that the extractor did not find.
- Never skip a call because the session was a harvest, a subagent, or “already classified.”
- Never reuse a previous — always classify this run from the extractor output.
- User-facing output is the HTML report + a short numeric summary.