Website Scraping and API Automation

Feature Description

• Analyze and crawl website structures
• Call and test REST/GraphQL APIs
• Create automated crawler scripts
• Data parsing and cleaning
• Handle anti-crawling mechanisms
• Scheduled tasks and data storage

Usage Scenarios

• "Scrape product information from this website"
• "Help me call this API and parse the returned data"
• "Create a script to regularly scrape news"
• "Analyze the API interface documentation of this website"
• "Bypass the anti-crawling restrictions of this website"

Technology Stack

Python Crawlers

• requests: HTTP request library
• BeautifulSoup4: HTML parsing
• Scrapy: Professional crawler framework
• Selenium: Browser automation
• Playwright: Modern browser automation

JavaScript Crawlers

• axios: HTTP client
• cheerio: Server-side jQuery
• puppeteer: Chrome automation
• node-fetch: Fetch API

Workflow

1. Target Analysis:
   • Check website structure and data locations
   • Analyze API interfaces and authentication methods
   • Evaluate anti-crawling mechanisms
2. Solution Design:
   • Select appropriate technology stack
   • Design data extraction strategies
   • Plan error handling and retry mechanisms
3. Script Development:
   • Write crawler code
   • Implement data parsing logic
   • Add logging and monitoring
4. Testing and Optimization:
   • Verify data accuracy
   • Optimize performance and stability
   • Handle edge cases

Best Practices

• Comply with robots.txt rules
• Set reasonable request intervals
• Use User-Agent and request headers
• Implement error retry mechanisms
• Data deduplication and verification
• Use proxy pools (if needed)
• Save raw data and logs

Common Scenario Examples

1. Simple Web Scraping

import requests
from bs4 import BeautifulSoup

def scrape_website(url):
    headers = {'User-Agent': 'Mozilla/5.0'}
    response = requests.get(url, headers=headers)
    soup = BeautifulSoup(response.text, 'html.parser')

    # 提取数据
    data = []
    for item in soup.select('.product'):
        data.append({
            'title': item.select_one('.title').text,
            'price': item.select_one('.price').text
        })
    return data

2. API Calling

import requests

def call_api(endpoint, params=None):
    headers = {
        'Authorization': 'Bearer YOUR_TOKEN',
        'Content-Type': 'application/json'
    }
    response = requests.get(endpoint, headers=headers, params=params)
    return response.json()

3. Dynamic Web Scraping

from selenium import webdriver
from selenium.webdriver.common.by import By

def scrape_dynamic_page(url):
    driver = webdriver.Chrome()
    driver.get(url)

    # 等待页面加载
    driver.implicitly_wait(10)

    # 提取数据
    elements = driver.find_elements(By.CLASS_NAME, 'item')
    data = [elem.text for elem in elements]

    driver.quit()
    return data

Anti-Crawling Countermeasures

• Request Header Spoofing: Simulate real browsers
• Proxy Rotation: Use proxy pools
• Captcha Handling: OCR or third-party services
• Cookie Management: Maintain session state
• Request Frequency Control: Avoid triggering restrictions
• JavaScript Rendering: Use Selenium/Playwright

Data Storage Solutions

• CSV/Excel: Simple data export
• JSON: Structured data storage
• Databases: MySQL, PostgreSQL, MongoDB
• Cloud Storage: S3, OSS
• Data Warehouses: For large-scale data analysis