archive-crawler — The Universal Archivist

Convention: see conventions/quality.md for citation rules, exact-phrasing requirements when capturing the user's reactions, and back-link enforcement.

Convention: see _brain-filing-rules.md — this skill is schema-generic: it reads the user's filing rules from the rules JSON instead of hardcoding any specific era / archive layout.

Safety gate (REQUIRED, no exceptions)

archive-crawler.scan_paths:

gbrain.yml

# gbrain.yml — the allow-list is mandatory
archive-crawler:
  scan_paths:
    - ~/Documents/writing/
    - ~/Dropbox/Archive/
    - /mnt/backup/old-letters/
  # Optional deny-list inside the allow-list:
  # deny_paths:
  #   - ~/Documents/finances/
  #   - ~/Documents/medical/

scan_paths

archive-crawler: refusing to run. No `archive-crawler.scan_paths:` allow-list
in gbrain.yml. Add explicit paths the agent is permitted to scan, then re-run.
This is a safety fence — the agent will not infer what's safe to read.

src/core/storage-config.ts

db_tracked

db_only

What this is

.mbox

Concepts

Source

• type:

  local

  |

  dropbox

  |

  backblaze

  |

  gmail-takeout

  |

  mbox

  |

  pst

• root: filesystem path, Dropbox path, B2 prefix, mbox path
• manifest: a brain page tracking progress at

  projects/<archive-slug>/STATUS.md

Manifest

1. Tree inventory — folders / files / sizes / types
2. Triage status — each item:

   ⬜ unseen

   /

   👀 reviewed

   /

   ✅ ingested

   /

   ⏭️ skip

   /

   🔥 high-signal

3. User reactions — exact quotes when they react (per conventions/quality.md exact-phrasing rule)
4. Priority queue — what to explore next, ranked
5. Session log — timestamped record of what was shown per session

Gold filter

Protocol

Phase 1: Inventory

1. Confirm scan_paths is set (safety gate). Exit if not.
2. Map the tree — list folders + files + sizes + date ranges.
3. Classify folders — group by likely content type (writing, email, code, photos, docs, system).
4. Create manifest — write

   projects/<archive-slug>/STATUS.md

   with the full inventory.
5. Propose priority queue — rank folders by likely gold density.
6. Present to user — show the map and proposed order. Let them override.

Phase 2: Crawl

1. Read before showing — open each candidate file, apply the gold filter, skip noise.
2. Show one at a time — present gold items individually for review.
3. Capture exact reaction — track the user's response in the manifest using their exact words (per conventions/quality.md).
4. Ingest if worth keeping — create a brain page immediately.
5. Update manifest — mark item status after each interaction.
6. Never re-show — check the manifest before presenting anything.

Phase 3: Ingest

_brain-filing-rules.md

• User's own writing / ideas / origin-story content →

  originals/<slug>.md

• Reflections / personal-life content →

  personal/<slug>.md

• Product / business ideas →

  ideas/<slug>.md

• Letters or threads about a specific person →

  people/<person>/timeline

  back-link plus the letter at

  personal/<slug>.md

  or

  originals/<slug>.md

originals/archive/

originals/yc-era/

_brain-filing-rules.json

---
title: "[Title or first line]"
type: original
source_type: "[local|dropbox|backblaze|gmail-takeout|mbox|pst]"
source_path: "[path within the allow-listed scan_paths]"
date: "YYYY-MM-DD"  # date from the file metadata or content
people: ["person-1", "person-2"]
tags: ["tag-1", "tag-2"]
---

# [Title]

[Summary: what it is, when it's from, why it matters]

**User's reaction:** [exact quote, no paraphrasing]

## Context

[Cross-links to people, concepts, projects.]

---

[Raw source material below the line — full text]

File-type handlers

Plain text / HTML / Markdown

.mbox

(email archives)

import mailbox
mbox = mailbox.mbox('/path/to/file.mbox')
for msg in mbox:
    body = ''
    if msg.is_multipart():
        for part in msg.walk():
            if part.get_content_type() == 'text/plain':
                body = part.get_payload(decode=True).decode('utf-8', errors='replace')
                break
    else:
        body = msg.get_payload(decode=True).decode('utf-8', errors='replace')
    # Apply gold filter

.doc

/

.docx

# .docx (modern)
python3 -c "
import zipfile, xml.etree.ElementTree as ET
with zipfile.ZipFile('/path/to/file.docx') as z:
    tree = ET.parse(z.open('word/document.xml'))
    print(''.join(t.text or '' for t in tree.iter('{http://schemas.openxmlformats.org/wordprocessingml/2006/main}t')))
"

# .doc (legacy, requires antiword or catdoc)
antiword /path/to/file.doc 2>/dev/null || catdoc /path/to/file.doc 2>/dev/null

.pst

(Outlook archives)

# Validate first; many PSTs are null bytes
python3 -c "
with open('/path/to/file.pst', 'rb') as f:
    print('Valid PST' if f.read(4) == b'!BDN' else 'CORRUPT/NULL')
"
# If valid:
readpst -o /tmp/pst-output /path/to/file.pst

.zip

/

.tar

/

.tar.gz

Images

Manifest template

---
title: "[Archive Name] — Ingestion Status"
type: project
created: YYYY-MM-DD
updated: YYYY-MM-DD
source_type: "[local|dropbox|...]"
scan_paths: ["paths from gbrain.yml"]
---

# [Archive Name] — Ingestion Status

## Source
- **Type:** [local|dropbox|...]
- **Allow-listed paths:** [from gbrain.yml]
- **Total files:** [N]
- **Total size:** [X GB]
- **Date range:** [earliest] — [latest]

## Inventory

### [Folder 1]
| Item | Type | Size | Status | Reaction |
|------|------|------|--------|----------|
| file1.txt | text | 2KB | ✅ ingested | 🔥 "exact quote" |
| file2.doc | doc | 15KB | ⏭️ skip | — |
| file3.html | html | 4KB | ⬜ unseen | — |

### [Folder 2]
...

## Priority Queue
1. [Highest priority — why]
2. [Next — why]
...

## Session Log

### YYYY-MM-DD — [Session topic]
- Reviewed: [list]
- Reactions: [exact quotes]
- Ingested: [brain pages created]
- Next: [what's queued]

Anti-Patterns

• ❌ Running without

  archive-crawler.scan_paths:

  set. Hard refusal. This is the safety contract — never bypass.
• ❌ Hardcoding era-specific filing paths (e.g.,

  originals/archive/

  ,

  originals/yc-era/

  ). Read filing rules at runtime instead.
• ❌ Re-showing items already marked in the manifest. The user's time is the scarcest resource.
• ❌ Paraphrasing reactions. Exact words only.
• ❌ Wrapping found content in lessons or takeaways. Let stories breathe.
• ❌ Skipping back-links when content references people / companies who have brain pages. Iron Law per conventions/quality.md.

Related skills

• skills/voice-note-ingest/SKILL.md

  — same exact-phrasing pattern for audio capture

• skills/idea-ingest/SKILL.md

  — single-link-or-article ingest with the same primary-subject filing rule

• skills/conventions/quality.md

  — citations, back-links, voice

Contract

• Routing matches the canonical triggers in the frontmatter.
• Output written under the directories listed in

  writes_to:

  (when applicable).
• Conventions referenced (

  quality.md

  ,

  brain-first.md

  ,

  _brain-filing-rules.md

  ) are followed.
• Privacy contract preserved: no real names, no fork-specific filesystem path literals, no upstream-fork references.

Output Format

test/skills-conformance.test.ts