data-base

Mental Model

Tool Selection

Anti-Patterns (NEVER)

• Don't scrape without checking robots.txt
• Don't overload servers (default: 1 req/sec)
• Don't scrape personal data without consent
• Don't use Chinese characters in output filenames (ASCII only)
• Don't forget to identify bot with User-Agent

Output Format

• JSON: Nested/hierarchical data
• CSV: Tabular data
• Filename:

  {source}_{timestamp}.{ext}

  (ASCII only, e.g.,

  news_20250115.csv

  )

Workflow

1. Ask: What data? Which sites? How much?
2. Select tool based on page type
3. Extract and save structured data
4. Deliver file path to user or pass to data-analysis

Python Environment

cd skills/data-base

if [ ! -f ".venv/bin/python" ]; then
    echo "Creating Python environment..."
    ./setup.sh
fi

.venv/bin/python your_script.py

References (load on demand)

references/REFERENCE.md

references/templates.md