Web to Markdown

Overview

• Headless browser (no visible window)
• Handles JavaScript-rendered content (SPAs, React, Vue, etc.)
• Batch processing multiple URLs → single file
• Self-contained (no MCP dependency)

Prerequisites

• Node.js/pnpm for running TypeScript scripts
• Dependencies installed:

  cd skills/web-to-markdown && pnpm install

• Playwright browsers will auto-install on first run

Usage

• "capture these pages as markdown"
• "save web content for documentation"
• "fetch and convert these webpages"
• "get markdown from these sites"
• "download and convert to markdown"

docs/web-captures/YYYYMMDD_HHMMSS.md

Workflow

Single Command

cd skills/web-to-markdown
pnpm tsx scripts/scrape-and-convert.ts <url1> [url2] [url3] ...

1. Creates timestamped output file with header
2. Launches headless Playwright browser
3. Scrapes each URL sequentially
4. Converts HTML → markdown (Turndown)
5. Appends to output file with formatted headers
6. Closes browser and reports summary

Examples

cd skills/web-to-markdown
pnpm tsx scripts/scrape-and-convert.ts https://example.com/docs

# Output: docs/web-captures/20251103_143052.md

cd skills/web-to-markdown
pnpm tsx scripts/scrape-and-convert.ts \
  https://example.com/guide \
  https://example.com/api \
  https://example.com/faq

# Output: docs/web-captures/20251103_143052.md (all 3 pages)

pnpm --filter @skills/web-to-markdown tsx scripts/scrape-and-convert.ts <urls...>

Output Format

# Web Captures - YYYY-MM-DD HH:MM:SS

Generated: YYYYMMDD_HHMMSS
URLs: N

---

## 📄 https://example.com/page1

[Converted markdown content...]

---

## 📄 https://example.com/page2

[Converted markdown content...]

---

Implementation Notes

• Pure functions (no classes except custom errors)
• Explicit error handling with typed errors:

  BrowserError

  ,

  FileError

  ,

  HtmlConversionError

• Small, focused functions
• Side effects isolated at edges
• CLI-style logging with chalk/ora

skills/web-to-markdown/
├── SKILL.md                    # This file (workflow instructions)
├── package.json                # pnpm workspace config
├── tsconfig.json               # TypeScript config
├── scripts/
│   ├── scrape-and-convert.ts   # Main CLI (Playwright + Turndown)
│   ├── html-to-markdown.ts     # Pure conversion function (Turndown wrapper)
│   └── convert-and-append.ts   # Legacy CLI (deprecated, kept for reference)
└── tests/
    └── html-to-markdown.test.ts # Unit tests

Error Handling

• Browser launch failure: Check Playwright installation, run

  pnpm exec playwright install chromium

• Page not found (404): Logs error, continues with other URLs
• Timeout (>30s): Reports slow page, continues to next URL
• Navigation error: Logs error, continues to next URL
• Conversion failure: Reports malformed HTML, skips page

Configuration

DEFAULT_CONFIG

scripts/scrape-and-convert.ts

const DEFAULT_CONFIG = {
  outputDir: 'docs/web-captures',
  timeout: 30000, // milliseconds
};

Performance

• Headless mode (no GUI overhead)
• Sequential processing (one URL at a time for stability)
• Browser reuse across URLs (faster than launching per-page)
• ~2-5 seconds per page (depends on site complexity)

Comparison with Alternatives

Troubleshooting

cd skills/web-to-markdown
pnpm exec playwright install chromium

• Increase timeout in

  DEFAULT_CONFIG

• Check network connectivity
• Some sites may block automated browsers (use Jina AI Reader alternative)

• Site may use heavy client-side rendering
• Try waiting longer (increase timeout)
• Check if site blocks headless browsers (User-Agent detection)

Notes

• One request = one file (all URLs aggregated)
• Handles JavaScript-rendered content (React, Vue, Angular, etc.)
• Headless by default (no visible browser window)
• Browser auto-installs on first run
• Ideal for documentation scraping and archival
• No external API dependencies