零基础也能掌握的完整爬虫技术路线,融合requests + BeautifulSoup + Selenium + 数据清洗全流程,提供可运行代码与避坑指南
选择爬虫工具需遵循三个原则:稳定性优先、生态成熟、文档完善。主流方案中:
新手推荐组合:requests + BeautifulSoup4,安装命令:
pip install requests beautifulsoup4 lxml
以下代码实现对目标网页的GET请求并提取标题与正文内容:
import requests
from bs4 import BeautifulSoup
# 目标URL(示例使用Python官网)
url = "https://www.python.org/"
# 设置请求头模拟浏览器
headers = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
}
# 发起HTTP请求
response = requests.get(url, headers=headers)
response.encoding = "utf-8" # 显式指定编码
# 解析HTML文档
soup = BeautifulSoup(response.text, "lxml")
# 提取关键信息
title = soup.find("title").get_text()
news_items = soup.select("ul.list-recent-events li")
print(f"页面标题:{title}")
print("\n最近动态:")
for item in news_items:
time = item.find("time").get_text()
title = item.find("a").get_text()
print(f" • {time} → {title}")
处理编码错误时,可采用以下策略:
推荐兼容性写法:
if response.encoding == "ISO-8859-1":
response.encoding = response.apparent_encoding
典型用法示例:
session = requests.Session()
session.headers.update(headers)
# 携带查询参数
params = {"q": "python爬虫", "page": 1}
res = session.get("https://example.com/search", params=params)
# POST表单提交
data = {"username": "user", "password": "pass"}
res = session.post("https://example.com/login", data=data)
# 处理JSON响应
json_data = res.json()
Scrapy是企业级爬虫框架,采用事件驱动模型,具备以下优势:
核心组件说明:
| 组件 | 作用 |
|---|---|
| Spider | 定义爬取逻辑与解析规则 |
| Engine | 控制数据流与事件触发 |
| Downloader | 处理HTTP请求与响应 |
| Item Pipeline | 清洗、验证、存储数据 |
针对JS渲染页面(如SPA应用),Selenium通过控制真实浏览器完成渲染:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
# 启动无头Chrome
options = webdriver.ChromeOptions()
options.add_argument("--headless")
driver = webdriver.Chrome(options=options)
try:
driver.get("https://dynamic-site.com")
# 等待元素加载完成
element = WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.CSS_SELECTOR, ".target"))
)
# 获取渲染后内容
content = driver.page_source
finally:
driver.quit()
注意事项:
Httpx融合同步/异步双模式,支持HTTP/2协议:
import httpx
# 同步模式
with httpx.Client() as client:
r = client.get("https://httpbin.org/get")
print(r.json())
# 异步模式
async def main():
async with httpx.AsyncClient() as client:
r = await client.get("https://httpbin.org/get")
print(r.json())
适用场景:
| 反爬类型 | 特征 | 应对方案 |
|---|---|---|
| Headers校验 | 检查User-Agent/Referer | 伪造完整请求头 |
| IP限制 | 频繁请求触发封禁 | 使用代理池+请求间隔 |
| Cookie验证 | 动态生成token | 模拟登录或逆向JS |
| JS混淆 | 动态生成数据或加密参数 | 抓包分析加密逻辑 |
| 滑块验证 | 行为验证机制 | 调用第三方打码平台 |
完整请求头应包含以下关键字段:
headers = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
"Accept-Language": "zh-CN,zh;q=0.9,en;q=0.8",
"Accept-Encoding": "gzip, deflate, br",
"Connection": "keep-alive",
"Referer": "https://www.google.com/",
"Sec-Fetch-Dest": "document",
"Sec-Fetch-Mode": "navigate",
"Sec-Fetch-Site": "none",
"Upgrade-Insecure-Requests": "1"
}
建议定期更换UA池,避免单一UA被标记:
ua_pool = [
"Mozilla/5.0 (Windows NT 10.0; Win64; x64)...",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)...",
"Mozilla/5.0 (X11; Linux x86_64)...",
"Mozilla/5.0 (iPhone; CPU iPhone OS 16_0)...",
]
以某网站加密参数a=xxx为例:
实战案例:
import execjs
# 从页面提取加密JS代码
with open("crypto.js", "r") as f:
js_code = f.read()
# 创建JS运行环境
ctx = execjs.compile(js_code)
# 调用加密函数
encrypted = ctx.call("encrypt", "raw_data")
| 存储方式 | 优势 | 适用场景 |
|---|---|---|
| CSV | 通用性强、Excel可直接打开 | 小型项目/临时数据 |
| JSON | 结构清晰、跨语言兼容 | API数据/配置存储 |
| SQLite | 零配置、单文件数据库 | 中型项目/本地存储 |
| MySQL | 支持复杂查询、事务处理 | 生产环境/高并发 |
CSV写入:
import csv
data = [
{"title": "爬虫教程", "author": "张三", "views": 1200},
{"title": "数据清洗", "author": "李四", "views": 850}
]
with open("data.csv", "w", newline="", encoding="utf-8-sig") as f:
writer = csv.DictWriter(f, fieldnames=data[0].keys())
writer.writeheader()
writer.writerows(data)
JSON写入:
import json
with open("data.json", "w", encoding="utf-8") as f:
json.dump(data, f, ensure_ascii=False, indent=2)
使用sqlite3模块创建表并插入数据:
import sqlite3
# 连接数据库(不存在则创建)
conn = sqlite3.connect("crawler.db")
cursor = conn.cursor()
# 创建数据表
cursor.execute("""
CREATE TABLE IF NOT EXISTS articles (
id INTEGER PRIMARY KEY AUTOINCREMENT,
title TEXT NOT NULL,
author TEXT,
views INTEGER,
created_at DATETIME DEFAULT CURRENT_TIMESTAMP
)
""")
# 插入数据(防SQL注入)
cursor.execute(
"INSERT INTO articles (title, author, views) VALUES (?, ?, ?)",
("爬虫实战", "Python技术社区", 2500)
)
conn.commit()
conn.close()
目标:抓取科技新闻网站的标题、作者、发布时间、阅读量
import requests
from bs4 import BeautifulSoup
import sqlite3
import time
from datetime import datetime
# 初始化数据库
conn = sqlite3.connect("news.db")
cursor = conn.cursor()
cursor.execute("""
CREATE TABLE IF NOT EXISTS tech_news (
id INTEGER PRIMARY KEY AUTOINCREMENT,
title TEXT NOT NULL,
author TEXT,
publish_time TEXT,
views INTEGER,
url TEXT UNIQUE,
crawled_at DATETIME DEFAULT CURRENT_TIMESTAMP
)
""")
def crawl_page(page):
url = f"https://example-tech.com/news?page={page}"
headers = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"
}
try:
res = requests.get(url, headers=headers, timeout=10)
res.raise_for_status()
res.encoding = "utf-8"
soup = BeautifulSoup(res.text, "lxml")
articles = soup.select(".news-item")
for item in articles:
title = item.select_one(".title").get_text().strip()
author = item.select_one(".author").get_text().strip()
pub_time = item.select_one(".time").get_text().strip()
views = int(item.select_one(".views").get_text().strip())
link = item.select_one("a")["href"]
# 插入数据(忽略重复)
cursor.execute("""
INSERT OR IGNORE INTO tech_news (title, author, publish_time, views, url)
VALUES (?, ?, ?, ?, ?)
""", (title, author, pub_time, views, link))
print(f"✅ 第{page}页抓取完成:{len(articles)}条")
return True
except Exception as e:
print(f"❌ 第{page}页失败:{str(e)}")
return False
# 主程序入口
if __name__ == "__main__":
for page in range(1, 4): # 抓取3页
crawl_page(page)
time.sleep(2) # 请求间隔避免封禁
conn.commit()
# 查询统计
cursor.execute("SELECT COUNT(*) FROM tech_news")
total = cursor.fetchone()[0]
print(f"\n📊 总计抓取:{total}条新闻")
conn.close()
数据入库前需进行清洗处理:
清洗函数示例:
def clean_views(raw):
"""处理阅读量格式:'1.2k' → 1200"""
if not raw:
return 0
raw = raw.strip().lower()
if raw.endswith("k"):
return int(float(raw[:-1]) * 1000)
elif raw.endswith("w"):
return int(float(raw[:-1]) * 10000)
try:
return int(raw)
except:
return 0
A:零基础可上手基础爬虫,但需掌握Python基础语法(变量/循环/函数),建议先学习10小时入门课程。核心是理解HTTP协议与DOM结构解析逻辑。
A:合法爬虫需遵守:
• 遵守robots.txt协议
• 控制请求频率(建议≥2秒/次)
• 不绕过访问控制机制
• 不用于商业侵权
重点参考《网络安全法》第27条及《民法典》第1034条关于个人信息保护规定。
A:常见原因:
① 目标网站启用JS动态加载 → 改用Selenium
② 请求被拦截 → 检查请求头与Cookie
③ 编码问题 → 确认response.apparent_encoding
④ 选择器错误 → 用浏览器开发者工具验证CSS选择器
A:建议实施:
• 添加异常处理(try-except)
• 设置请求超时(timeout=10)
• 实现重试机制(最多3次)
• 日志记录关键节点
• 使用代理IP池防封禁