Scrapy:专业级分布式爬虫框架
基于Twisted异步网络库构建,适用于大规模数据采集:
- 核心组件:Spider(爬虫逻辑)、Engine(引擎)、Scheduler(调度器)、Downloader(下载器)、Item Pipeline(数据处理)
- 中间件系统:Downloader Middleware处理请求/响应,Spider Middleware处理Item
- 自动限速:AutoThrottle中间件动态调整请求间隔
- Scrapy-Redis:支持分布式爬取,共享任务队列
典型项目结构:
myproject/
├── scrapy.cfg
└── myproject/
├── settings.py
├── middlewares.py
└── spiders/
├── __init__.py
└── example.py
Spider核心逻辑:
import scrapy
class NewsSpider(scrapy.Spider):
name = "news"
start_urls = ["https://example.com/news"]
def parse(self, response):
for article in response.css("div.article"):
yield {
"title": article.css("h2::text").get(),
"link": article.css("a::attr(href)").get(),
"date": article.css("time::text").get()
}
next_page = response.css("a.next::attr(href)").get()
if next_page:
yield response.follow(next_page, callback=self.parse)
Selenium:浏览器自动化测试
适用于处理JavaScript渲染页面、模拟用户操作(点击/滚动/输入):
- WebDriver支持:Chrome/Firefox/Edge等主流浏览器驱动
- 显式等待:WebDriverWait配合expected_conditions处理动态加载
- 无头模式:headless选项提升执行效率
- 性能瓶颈:资源占用高,不适合高并发采集
典型用法:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
options = webdriver.ChromeOptions()
options.add_argument("--headless")
options.add_argument("--disable-gpu")
driver = webdriver.Chrome(options=options)
driver.get("https://example.com")
try:
element = WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.CSS_SELECTOR, ".content"))
)
print(element.text)
finally:
driver.quit()
HTTPX:现代异步HTTP客户端
Requests的异步升级版,支持HTTP/2、并发请求:
- 同步/异步双模式:sync_client与async_client自由切换
- 连接池复用:提升高并发场景性能
- 同步兼容:API与Requests高度一致,迁移成本低
- HTTP/2支持:多路复用减少延迟
异步采集示例:
import asyncio
import httpx
async def fetch_urls(urls):
async with httpx.AsyncClient() as client:
tasks = [client.get(url) for url in urls]
responses = await asyncio.gather(*tasks)
return [r.json() for r in responses]
async def main():
urls = ["https://api.example.com/data1",
"https://api.example.com/data2"]
data = await fetch_urls(urls)
print(data)
asyncio.run(main())
Playwright:微软开源现代自动化工具
Chrome DevTools Protocol实现,支持浏览器自动化:
- 自动等待:无需手动等待,自动检测页面稳定状态
- 跨浏览器:Chromium/Firefox/WebKit统一API
- 网络拦截:mock API请求,绕过部分反爬
- 截图/录屏:支持页面级截图与视频录制
网络拦截示例:
from playwright.sync_api import sync_playwright
with sync_playwright() as p:
browser = p.chromium.launch(headless=True)
page = browser.new_page()
page.route("**/api/data", lambda route: route.continue_(
url="https://mock.example.com/data"
))
page.goto("https://example.com")
content = page.content()
browser.close()