indexion refactor — Codebase Refactoring
Detect and eliminate duplication at three levels — textual, structural, and conceptual —
using indexion's analysis commands, then verify SoT is enforced.
When to Use
- After adding a new abstraction (type, module, API layer)
- After introducing a new file format or I/O boundary
- When a fix required touching 3+ files for the same reason
- When a "guard" or "skip" was added to work around a structural problem
- When , , or similar filesystem errors appear from unexpected paths
- When extracting shared code across packages
- When cleaning up after a refactor (removing trivial wrapper functions)
- Periodic SoT health check on a codebase
Three Levels of Duplication
| Level | What it is | Tool | Example |
|---|
| Textual | Copy-pasted code blocks, identical functions | | copied across 5 modules |
| Structural | Same logic structure with different names | , | cross-package extraction candidates, trivial wrappers |
| Conceptual | Same domain concept implemented independently | + manual analysis | Three modules each deciding "is this file an archive?" |
Textual duplication is easy to find and fix. Conceptual duplication is the hardest and
most dangerous — it produces no copy-paste matches but means changing one concept requires
updating every scattered implementation.
Workflow
Phase 1: Clear textual duplication ()
Start with high-confidence matches and work down.
bash
# Step 1: Find 90%+ duplicates (high confidence)
indexion plan refactor --threshold=0.9 \
--include='*.mbt' --exclude='*_wbtest.mbt' \
--exclude='*moon.pkg*' --exclude='*pkg.generated*' \
cmd/indexion/
indexion plan refactor --threshold=0.9 \
--include='*.mbt' --exclude='*_wbtest.mbt' \
--exclude='*moon.pkg*' --exclude='*pkg.generated*' \
src/
Read the output in three sections:
| Section | What it finds | Action |
|---|
| Similar Files | Files with high overall similarity | Investigate for structural consolidation |
| Duplicate Code Blocks | Line-level identical code between files | Extract to or shared module |
| Function-Level Duplicates | Structurally similar functions (TF-IDF on bodies) | Unify into single SoT function |
Same-file duplicates (functions within one file at 90%+) are the highest-value
targets — easiest to fix, clearest wins. Example:
and
share 95% structure, extracted into
.
bash
# Step 2: Use grep to trace references before consolidating
indexion grep "TypeIdent:TfidfEmbeddingProvider" src/
indexion grep --semantic=name:is_whitespace src/
# Step 3: Fix, then re-run to confirm duplicates are gone
indexion plan refactor --threshold=0.9 --include='*.mbt' ...
# Step 4: Lower threshold and iterate
indexion plan refactor --threshold=0.85 --include='*.mbt' ...
| Option | Default | Description |
|---|
| 0.7 | Minimum similarity threshold |
| hybrid | Similarity: hybrid, tfidf, bm25, jsd, ncd |
| 0 | FDR correction (0=disabled) |
| raw | Output: raw, structured |
| md | Output: md, json, text, github-issue |
| -- | Project name (for structured style) |
| -- | Include pattern (repeatable) |
| -- | Exclude pattern (repeatable) |
| stdout | Output file path |
| kgfs | KGF specs directory |
What remains after cleanup (stop signals):
- Platform stubs ( / ) — intentional platform branching
- Type method similarity ( on different types) — different types, same pattern
- CLI command boilerplate ( functions) — @argparse API pattern, not duplication
- Semantic-but-different functions ( vs ) — different purpose
Phase 2: Extract cross-package shared code ()
After cleaning within each directory, find code that should be shared across packages.
bash
# Find overlap between two packages
indexion plan solid --from=src/a,src/b
# Specify extraction target
indexion plan solid --from=src/a,src/b --to=src/common
# Use tree edit distance for precise function-level matching
indexion plan solid --from=src/a,src/b --strategy=apted
# Higher threshold for stricter matching
indexion plan solid --from=src/a,src/b --threshold=0.95
# Filter files
indexion plan solid --from=src/a,src/b --include='*.mbt' --exclude='*_test.mbt'
| | |
|---|
| Scope | Internal duplication within directories | Cross-directory overlap |
| Goal | Consolidate within a codebase | Extract shared code into a new package |
| Input | | |
| Option | Default | Description |
|---|
| (required) | Source directories (comma-separated or repeatable) |
| -- | Target directory for extraction |
| -- | Rules file (.solidrc) |
| -- | Inline rule (repeatable) |
| 0.9 | Minimum similarity threshold |
| tfidf | Similarity: tfidf, apted, tsed |
| -- | Include pattern (repeatable) |
| -- | Exclude pattern (repeatable) |
| md | Output: md, json, github-issue |
| stdout | Output file path |
| kgfs | KGF specs directory |
Workflow:
- Run on each directory individually first to clean internal duplication
- Run
plan solid --from=dirA,dirB
- Extract shared code following the plan's recommendations
- Use
indexion grep "TypeIdent:SharedType"
Phase 3: Remove unnecessary wrappers ()
After consolidation, clean up trivial delegation functions that add indirection without value.
bash
# Step 1: Quick check
indexion grep --semantic=proxy src/
# Step 2: Detailed report
indexion plan unwrap --include='*.mbt' --exclude='*_wbtest.mbt' \
--exclude='*moon.pkg*' --exclude='*pkg.generated*' src/
# Step 3: Preview changes (safe — no files modified)
indexion plan unwrap --dry-run --include='*.mbt' --exclude='*_wbtest.mbt' \
--exclude='*moon.pkg*' --exclude='*pkg.generated*' src/
# Step 4: Apply fixes
indexion plan unwrap --fix --include='*.mbt' --exclude='*_wbtest.mbt' \
--exclude='*moon.pkg*' --exclude='*pkg.generated*' src/
# Step 5: Run tests
moon test --target native
What gets detected: Functions whose body is a single function call with
all arguments forwarded as simple identifiers — no control flow, no transforms.
moonbit
// Detected (default) — trivial delegation
fn matches_pattern(text : String, pat : String) -> Bool {
@glob.glob_match(text, pat)
}
// Excluded by default (use --all to include)
fn length(self : MyList) -> Int {
self.items.length() // self-delegation (encapsulation)
}
fn emit(value : String) -> Action {
Emit(value) // bare constructor
}
| Mode | Flag | Description |
|---|
| Report | (default) | List wrappers found |
| Preview | | Show all edits without modifying files |
| Fix | | Apply edits to files |
| Option | Default | Description |
|---|
| -- | Preview edits |
| -- | Apply edits |
| -- | Include self-delegation and bare constructor wrappers |
| -- | Include patterns |
| -- | Include bare constructor wrappers |
| -- | Include pattern (repeatable) |
| -- | Exclude pattern (repeatable) |
| md | Output: md, json, text |
| stdout | Output file path |
| kgfs | KGF specs directory |
Review before removing:
- Platform wrappers (FFI, ) are abstraction layers, not accidental indirection
- Public API wrappers used by external packages — removing them is a breaking change
- Always first
Phase 4: Detect concept-level duplication ( + analysis)
This is the hardest level. Textual and structural tools won't find it because the
code is different — but the concept is the same.
bash
# Find which files share vocabulary (= work in the same concept domain)
indexion explore --threshold=0.4 \
--include='*.mbt' --exclude='*_wbtest.mbt' \
--exclude='*moon.pkg*' --exclude='*pkg.generated*' \
src/ cmd/
Files at 40-60% similarity without structural duplication are concept neighbors —
they use the same terms because they deal with the same domain.
For each high-similarity pair, ask: "What concept do they share, and who owns it?"
bash
# Inspect shared vocabulary with tree structure comparison
indexion explore file_a.mbt file_b.mbt --threshold=0 --strategy=apted
Common patterns of concept leakage:
| Symptom | Concept leaked | Fix |
|---|
| Both files call then | "Determine if X and configure Y" | Extract into the module that owns X |
Both files @fs.read_file_to_string(path)
| "Read file content" | Pass content as argument, don't re-read |
| Both files then | "List sibling files" | Centralize directory walking into pipeline |
Multiple if is_virtual_path(x) { skip }
| "Real vs virtual path" | Make the type system prevent virtual paths from reaching here |
Both files buf.write_string("\n"); buf.write_string(x)
| "Join text entries" | Extract into the owning module |
Phase 5: Consolidate into SoT
The module that defines the concept should be the only one that implements the logic.
Rules:
- One concept, one module, one function. If "extract text from archive" appears in , , and , it belongs in only.
- Callers receive results, not ingredients. Don't export + + separately. Export
try_extract_archive_text(path, spec) -> String?
- Guards are symptoms, not fixes.
if is_virtual_path(x) { skip }
- Re-reading from disk what's already in memory is a concept leak. If holds the text, no downstream code should call
@fs.read_file_to_string(file.path)
Phase 6: Verify
bash
# Confirm textual duplication is gone
indexion plan refactor --threshold=0.9 \
--include='*.mbt' --exclude='*_wbtest.mbt' \
--exclude='*moon.pkg*' --exclude='*pkg.generated*' \
src/ cmd/indexion/
# Confirm concept similarity is reduced
indexion explore file_a.mbt file_b.mbt --threshold=0
# Confirm wrappers are cleaned up
indexion plan unwrap --include='*.mbt' --exclude='*_wbtest.mbt' \
--exclude='*moon.pkg*' --exclude='*pkg.generated*' src/
# Run tests
moon test --target native
After SoT consolidation:
- Textual similarity between the concept owner and its callers drops
- Callers become shorter (one API call instead of multi-step logic)
- The concept owner may grow, but it's the single place to change
Phase 7: Prove non-recurrence with tests
Write a test that structurally prevents the old pattern from recurring:
moonbit
test "SoT: SupportedFile.path is always a real filesystem path" {
// Create an archive, run load_supported_file_info
// Assert: no path contains "!/"
// Assert: every path passes @fs.path_exists
}
The test doesn't check behavior — it checks the SoT invariant.
Red Flags
"I need to add a guard here"
If you're adding
if is_special_case(x) { skip }
to a function that shouldn't receive
special cases,
the problem is upstream. The function's caller should never pass that value.
"It works but prints errors to stderr"
Stderr messages from C runtime (
opendir: No such file or directory
) mean invalid data
reached a system call.
absorbs the error, but
already printed.
The only fix is preventing invalid data from reaching the call.
"I'll fix it in each command separately"
If the same fix is needed in explore, search, grep, reconcile, plan documentation...
the fix belongs in the shared pipeline, not in each command.
"The similarity is just shared vocabulary, not real duplication"
40-60% TF-IDF similarity between modules that aren't supposed to share concepts is a
warning. The vocabulary match IS the signal.
Quick Reference: Which Command When
| Question | Command |
|---|
| "What files are similar?" | |
| "What exactly is duplicated?" | plan refactor --threshold=0.9
|
| "What code overlaps between packages A and B?" | |
| "Which functions are trivial wrappers?" | or |
| "What concept do these files share?" | explore file_a file_b --threshold=0 --strategy=apted
|
| "Has the duplication been fixed?" | Re-run with same threshold |