Web Scraping with Python Skill

1. Scraper Building — Design and implement web scrapers with idiomatic, production-ready patterns
2. Scraper Review — Analyze existing scrapers against the book's practices and recommend improvements

How to Decide Which Mode

• If the user asks to build, create, scrape, extract, crawl, or collect data → Scraper Building
• If the user asks to review, audit, improve, debug, optimize, or fix a scraper → Scraper Review
• If ambiguous, ask briefly which mode they'd prefer

Mode 1: Scraper Building

Step 1 — Understand the Requirements

• What target? — Single page, single domain, multiple domains, API endpoints?
• What data? — Text, tables, images, documents, forms, dynamic JavaScript content?
• What scale? — One-off extraction, recurring crawl, large-scale parallel scraping?
• What challenges? — Login required, JavaScript rendering, rate limiting, anti-bot measures?

Step 2 — Apply the Right Practices

references/practices-catalog.md

Step 3 — Follow Web Scraping Principles

1. Respect robots.txt — Always check and honor robots.txt directives; be a good citizen of the web
2. Identify yourself — Set a descriptive User-Agent string; consider providing contact info
3. Rate limit requests — Add delays between requests (1-3 seconds minimum); never hammer servers
4. Handle errors gracefully — Catch connection errors, timeouts, HTTP errors, and missing elements
5. Use sessions wisely — Reuse HTTP sessions for connection pooling and cookie persistence
6. Parse defensively — Never assume HTML structure is stable; use multiple selectors as fallbacks
7. Store raw data first — Save raw HTML/responses before parsing; enables re-parsing without re-scraping
8. Validate extracted data — Check for None/empty values; verify data types and formats
9. Design for re-runs — Make scrapers idempotent; track what's already been scraped
10. Stay legal and ethical — Understand applicable laws (CFAA, GDPR); respect Terms of Service

Step 4 — Build the Scraper

• Production-ready — Include error handling, retries, logging, rate limiting from the start
• Configurable — Externalize URLs, selectors, delays, credentials; use config files or arguments
• Testable — Write unit tests for parsing functions; integration tests for full scrape flows
• Observable — Log page fetches, items extracted, errors encountered, timing stats
• Documented — README with setup, usage, target site info, legal notes

1. Approach identification — Which chapters/concepts apply and why
2. Target analysis — Site structure, pagination, authentication needs, JS rendering
3. Implementation — Production-ready code with error handling and rate limiting
4. Storage setup — How and where data is stored (CSV, database, files)
5. Monitoring notes — What to watch for (site changes, blocks, data quality)

Scraper Building Examples

User: "Scrape product listings from an e-commerce category page"

Apply: Ch 1 (fetching pages), Ch 2 (parsing product elements),
       Ch 3 (pagination/crawling), Ch 6 (storing to CSV/DB)

Generate:
- requests + BeautifulSoup scraper
- CSS selector-based product extraction
- Pagination handler following next-page links
- CSV or database storage with schema
- Rate limiting and error handling

User: "Extract data from a React single-page application"

Apply: Ch 11 (Selenium, headless browser), Ch 2 (parsing rendered HTML),
       Ch 14 (avoiding detection), Ch 15 (testing)

Generate:
- Selenium WebDriver with headless Chrome
- Explicit waits for dynamic content loading
- JavaScript execution for scrolling/interaction
- Data extraction from rendered DOM
- Headless browser configuration

User: "Scrape data from a site that requires login"

Apply: Ch 10 (forms, sessions, cookies), Ch 14 (headers, tokens),
       Ch 6 (data storage)

Generate:
- Session-based login with CSRF token handling
- Cookie persistence across requests
- POST request for form submission
- Authenticated page navigation
- Session expiry detection and re-login

User: "Build a crawler to scrape thousands of pages from multiple domains"

Apply: Ch 5 (Scrapy framework), Ch 4 (crawl models),
       Ch 16 (parallel scraping), Ch 14 (avoiding blocks)

Generate:
- Scrapy spider with item definitions and pipelines
- CrawlSpider with Rule and LinkExtractor
- Pipeline for database storage
- Settings for concurrent requests, delays, user agents
- Middleware for proxy rotation

Mode 2: Scraper Review

references/review-checklist.md

Review Process

1. Fetching scan — Check Ch 1, 10, 11: HTTP method, session usage, JS rendering needs, authentication
2. Parsing scan — Check Ch 2, 7: selector quality, defensive parsing, edge case handling
3. Crawling scan — Check Ch 3-5: URL management, deduplication, pagination, depth control
4. Storage scan — Check Ch 6: data format, schema, duplicates, file management
5. Resilience scan — Check Ch 14-16: error handling, retries, rate limiting, parallel safety
6. Ethics scan — Check Ch 17-18: robots.txt, legal compliance, identification, respectful crawling
7. Quality scan — Check Ch 8, 15: data cleaning, testing, validation

Review Output Format

## Summary
One paragraph: overall scraper quality, pattern adherence, main concerns.

## Fetching & Connection Issues
For each issue (Ch 1, 10-11):
- **Topic**: chapter and concept
- **Location**: where in the code
- **Problem**: what's wrong
- **Fix**: recommended change with code snippet

## Parsing & Extraction Issues
For each issue (Ch 2, 7):
- Same structure

## Crawling & Navigation Issues
For each issue (Ch 3-5):
- Same structure

## Storage & Data Issues
For each issue (Ch 6, 8):
- Same structure

## Resilience & Performance Issues
For each issue (Ch 14-16):
- Same structure

## Ethics & Legal Issues
For each issue (Ch 17-18):
- Same structure

## Testing & Quality Issues
For each issue (Ch 9, 15):
- Same structure

## Recommendations
Priority-ordered from most critical to nice-to-have.
Each recommendation references the specific chapter/concept.

Common Web Scraping Anti-Patterns to Flag

• No error handling on requests → Ch 1, 14: Wrap requests in try/except; handle ConnectionError, Timeout, HTTPError
• Hardcoded selectors without fallbacks → Ch 2: Use multiple selector strategies; check for None before accessing attributes
• No rate limiting → Ch 14: Add time.sleep() between requests; respect server resources
• Missing User-Agent header → Ch 14: Set a descriptive User-Agent; rotate if needed for scale
• Not using sessions → Ch 10: Use requests.Session() for cookie persistence and connection pooling
• Ignoring robots.txt → Ch 18: Parse and respect robots.txt before crawling
• No URL deduplication → Ch 3: Track visited URLs in a set; normalize URLs before comparing
• Using regex to parse HTML → Ch 2: Use BeautifulSoup or lxml, not regex, for HTML parsing
• Not handling JavaScript content → Ch 11: If data loads via Ajax, use Selenium or find the underlying API
• Storing data without validation → Ch 6, 8: Validate and clean data before storage; handle encoding
• No logging → Ch 5: Log requests, responses, errors, extracted items; track progress
• Sequential when parallel is needed → Ch 16: Use threading/multiprocessing for large-scale scraping
• Ignoring encoding issues → Ch 7, 8: Handle UTF-8, detect encoding, normalize Unicode
• No tests for parsers → Ch 15: Write unit tests with saved HTML fixtures; test selector robustness
• Credentials in code → Ch 10: Use environment variables or config files for login credentials
• Not storing raw responses → Ch 6: Save raw HTML for re-parsing; don't rely only on extracted data

General Guidelines

• BeautifulSoup for simple scraping, Scrapy for scale — Match the tool to the complexity
• Check for APIs first — Many sites have APIs (documented or undocumented) that are easier than scraping
• Respect the site — Rate limit, identify yourself, follow robots.txt, check ToS
• Parse defensively — HTML structure changes; always handle missing elements gracefully
• Test with saved pages — Save HTML fixtures and test parsers offline; reduces requests and enables CI
• Clean data early — Normalize strings, handle encoding, strip whitespace at extraction time
• For deeper practice details, read

  references/practices-catalog.md

  before building scrapers.
• For review checklists, read

  references/review-checklist.md

  before reviewing scrapers.