Scrapling

When to Use

• Extracting data from websites that change their HTML structure frequently
• Bypassing anti-bot protections (Cloudflare Turnstile, WAFs, fingerprinting)
• High-performance web data collection at scale
• Replacing brittle BeautifulSoup/Scrapy selectors with adaptive element tracking
• AI-assisted data extraction via the built-in MCP server (Claude/Cursor integration)
• Interactive web exploration and debugging via the CLI shell

How It Works

1. Smart Fetching — Three fetcher tiers for different protection levels:

   • Fetcher

     — Fast HTTP with TLS fingerprinting and stealth headers

   • StealthyFetcher

     — Modified Firefox with fingerprint spoofing, bypasses Cloudflare

   • DynamicFetcher

     — Full Playwright browser automation in stealth mode
2. Adaptive Parsing — Tracks elements via similarity algorithms. When a website changes its structure, Scrapling automatically relocates the elements you need instead of breaking.
3. Developer Tools — Interactive IPython shell, CLI extraction commands, curl-to-Scrapling conversion, and an MCP server for AI agent integration.

Quick Start

pip install "scrapling[all]" && scrapling install

from scrapling.fetchers import StealthyFetcher

url = 'https://example.com'
page = StealthyFetcher.get(url, headless=True)  # Adaptive stealth fetching
products = page.css('.product', adaptive=True)   # Survives site changes

for product in products:
    print(product.css_first('.title').text)
    print(product.css_first('.price').text)

Features

• Adaptive element tracking — Auto-relocates elements after site structure changes via similarity algorithms
• Three fetcher tiers — Static HTTP, stealth Firefox, full Playwright browser automation
• Anti-bot bypass — Defeats Cloudflare Turnstile, WAFs, TLS fingerprinting, and browser detection
• Blazing fast — Outperforms Parsel, Scrapy, and BeautifulSoup in benchmarks
• CSS + XPath selectors — Plus text/regex search, BeautifulSoup-style navigation, auto-selector generation
• Async support — All fetchers support async/await
• Persistent sessions — FetcherSession, StealthySession, DynamicSession (sync and async)
• Interactive shell —

  scrapling shell

  for live exploration, curl conversion, browser previews
• CLI extraction —

  scrapling extract get URL output.md --css-selector

• MCP server — AI integration for Claude, Cursor, and other MCP-compatible agents
• Docker ready —

  docker pull pyd4vinci/scrapling

  (includes all browsers)

Performance

Installation Options

pip install scrapling                     # Core parser only
pip install "scrapling[fetchers]"         # + browser fetchers
scrapling install                         # Install browser engines
pip install "scrapling[all]"              # Everything

Source

• Repository: github.com/D4Vinci/Scrapling (8k+ stars)
• Documentation: scrapling.readthedocs.io
• Author: D4Vinci