Skill Progressive Disclosure Design

Triggering vs. disclosure: separate these first

description

run_loop.py

skill-creator

references/*

scripts/*

Default: do not split

• SKILL.md exceeds ~400 lines and content has natural branches.
• Empirical evidence (eval transcripts) shows the model wasting context on irrelevant sections.
• Specific content is large and only needed in narrow conditions.

• Single context load, no router prose to maintain.
• No tool-call overhead from reading references.
• No risk of the model loading the wrong reference or skipping a needed one.
• Easier to maintain: one file, one source of truth.
• Better for highly interconnected content where context is global.
• Easier for human reviewers to read end-to-end.

• Every invocation pays the full token cost, even when only 10% of the content is relevant.
• Does not scale past ~500 lines without degrading the model's ability to find what matters.
• No mechanism to gate rare or niche content.
• All content must justify its always-loaded status.
• Maintenance gets harder as the file grows.

Three split axes that work

1. Variant branch

references/<variant>.md

my-skill/
├── SKILL.md               # decision tree + shared steps
└── references/
    ├── variant-a.md
    ├── variant-b.md
    └── variant-c.md

• Each invocation loads only the matching variant; large savings when variants are big.
• Variants evolve independently, simplifying maintenance.
• Adding a new variant does not bloat existing content.
• Mental model is easy: select one path based on input.
• Maps cleanly to user intent that already mentions the variant.

• Requires routing logic in SKILL.md, eating back some of the line savings.
• Cross-cutting changes touch every variant file, multiplying effort.
• Risk of treatments diverging across variants over time.
• If user intent is ambiguous, the model may load multiple variants and lose the savings.
• If variants share more than ~60% of their content, the abstraction breaks down.

2. Workflow vs. reference data

references/

• Highest leverage of all splits: lookups are narrow, the model reads one entry.
• Natural conceptual boundary (procedure vs. data).
• Reference can grow large without affecting per-invocation cost.
• Adding new reference entries does not touch the workflow.
• Reference data can often be machine-generated and regenerated.

• The model must know what to look up before reading. Pointer must encode lookup keys explicitly.
• Fails when the workflow needs to weave reference data inline rather than at discrete points.
• Splits content that is conceptually unified, harder for human readers.
• The model may miss broader context that lives only in the reference.
• Lookup data that is small (under ~50 lines total) is rarely worth splitting.

3. Depth tier (common path vs. edge cases)

references/edge-cases.md

If you see X, Y, or Z, stop and read

references/edge-cases.md

before continuing.

• Common path stays minimal, fast, cheap.
• Edge cases can be exhaustive without polluting every invocation.
• Easy to extend edge-case coverage without touching the common path.
• Mirrors how experts work: defaults first, exceptions on demand.

• The load condition must be sharp and observable from user input. Most edge cases do not satisfy this.
• Vague conditions cause either always-loading (waste) or never-loading (dead weight).
• Edge cases get less testing because evals naturally cluster on common queries.
• The model may follow the common path past a point where it should have escalated.
• The 80/20 estimate is often wrong; what looked like an edge case turns out to be common.

Splits that do not work

Topic-based splits where invocations do not cluster by topic

unit.md

integration.md

mocks.md

• Conceptually clean, mirrors how a human would organize documentation.
• Easy to navigate as a maintainer.
• Plausibly reduces context per invocation.

• Real tasks span 2-3 topics, forcing multiple loads per invocation.
• Cross-topic concerns get duplicated or fragmented.
• The savings are theoretical, not empirical.

Splitting to hit a line target without a real branching condition

• A heuristic ("keep SKILL.md under 400 lines") feels like a clean rule to satisfy.
• Splitting feels like progress.

• Without a branching condition, references load in parallel or always, providing no savings.
• Adds router prose to SKILL.md, often making the total content longer.

Rare-but-critical content in references/

• The content is large or specialized.
• Moving it out of SKILL.md feels like good hygiene.

• References are optional by design; the model may skip them.
• If the content is critical, it must be loaded reliably, which means SKILL.md.
• "Rare" and "critical" together is usually a sign the skill is doing two jobs and should be two skills.

Cosmetic splits (Examples, Notes, Tips files)

• Reduces visual clutter in SKILL.md.
• Feels like good organization.

• No load condition: either always loaded (wasted tool call) or never loaded (dead content).
• Implies an importance hierarchy that does not exist at runtime.
• Frequently hides content from the model that needs it.

Pointer hygiene

• Name the user-visible signal that triggers the load. "If the user mentions snapshot tests" not "for testing concerns".
• One sentence per pointer. Do not summarize the reference content in SKILL.md.
• Encode the load condition in the filename.

  go126-simd.md

  not

  advanced.md

  .
• Top-of-file table of contents for any reference over 300 lines.
• If two references are co-loaded in most runs, merge them.

• Wrong-load rate drops sharply.
• Filename encodes load condition, self-documenting for future maintainers.
• Forces upfront clarity about when each reference is needed.
• Makes architecture evals easier to interpret.

• Some content has no crisp trigger; rules force awkward formulations.
• Filenames become long and awkward.
• Requires discipline; easy to drift over time.
• Can over-constrain useful loads when the trigger condition is genuinely fuzzy.

Use scripts/ before references/

scripts/

references/

• Zero context cost for execution.
• Deterministic, repeatable output.
• Reusable across invocations without re-reading.
• Can be unit tested independently.
• Often faster than prose-driven generation by the model.

• Requires the runtime to support script execution; not all environments do.
• Less flexible than letting the model reason over prose.
• Harder to handle unanticipated edge cases without code changes.
• Adds a maintenance burden: code in the skill needs to keep working.
• Users cannot easily customize behavior without editing the script.
• Failure modes are sharper: script errors stop the workflow.

Decision checklist

1. Does this content have a sharp, observable load condition the model can detect from user input?
2. Will splitting actually reduce context, accounting for the router prose added to SKILL.md?
3. Is this reference data (lookup) or procedural (sequence)? Procedural content usually stays.
4. Could a script handle this deterministically instead?
5. Across realistic invocations, what fraction of runs would load this file? Below 20%, inline or delete — rarely-loaded references rarely justify the routing overhead. 20–80% is the split sweet spot. Above 80%, promote into SKILL.md — the routing cost exceeds the load savings.

Evaluating skill architecture

• Catches dead references, dead SKILL.md sections, and mis-routed content.
• Quantifies whether a split actually saved tokens or just looked clean.
• Reveals real load patterns that intuition misses.
• Forces the eval set to cover all declared paths, surfacing dead paths.
• Compounds with output evals to catch regressions across both axes.

• Requires harness setup beyond standard output evals.
• Eval-set design for path coverage takes work.
• Metrics need calibration per-skill (thresholds vary with cost profile).
• Output evals are still required; this adds to total iteration cost.
• Easy to over-optimize for token cost at the expense of output quality.

Eval set design for architecture

• One query per declared variant (if the skill uses variant-branch splits).
• One query per edge-case branch (if depth-tier splits exist).
• One query per major lookup category (if reference-data splits exist).
• One query that should hit the common path only and load zero references.
• 2-3 off-topic queries that should not trigger the skill at all (also tests the description).

Instrumentation

1. Full transcript including every tool call.
2. Which

   references/*

   files were read (parse

   view

   calls on paths inside the skill directory).
3. Whether

   scripts/*

   were invoked.
4. Total tokens and wall time.
5. The output (for the parallel output-quality eval).

transcript.json

loads.json

timing.json

Metrics per reference file

references/*.md

• Load rate: fraction of runs that read it.
• Co-occurrence: for each other reference, fraction of runs that loaded both.
• Use rate when loaded: of the runs that loaded it, did the content visibly inform the output (cited content, applied procedure, used schema)? Inspect transcripts.
• Re-read rate: fraction of runs that loaded the same file twice.

Metrics for the skill overall

• Median and p95 tokens per invocation, with and without references.
• SKILL.md utilization: read transcripts and identify sections of SKILL.md the model never references in any run. Strong candidates for deletion.
• Path coverage: did every declared path get hit by at least one query?

Decision rules

Comparing two architectures

1. Run the identical eval set against both versions.
2. Run output-quality evals on both. Confirm no regression. If quality drops, the architecture change is a loss regardless of token savings.
3. Compare median tokens, p95 tokens, and median time per run.
4. Compare path coverage: does each version reliably reach the same outputs through the expected paths?

What the metrics will not tell you

• Whether the SKILL.md prose is clear. Read transcripts for confused tool calls and dead-end attempts.
• Whether the description triggers correctly. That is a separate eval (use

  run_loop.py

  from the

  skill-creator

  skill).
• Whether content placement matches user mental models. Subjective; review with a human.

Output when advising

1. Diagnose first. Is this a triggering question or a disclosure question?
2. Quote relevant content from the existing SKILL.md (or the user's description of it) before recommending.
3. Propose the minimum viable split. Resist splitting into more files than necessary.
4. For each proposed reference file, write the exact pointer sentence that would go in SKILL.md.
5. Surface the trade-offs explicitly. Use the pros/cons in this skill as the model for how to present a recommendation.
6. If unsure whether a split helps, recommend instrumentation (eval the skill, read transcripts) before committing.