在量化投资浪潮席卷全球的今天,数据已成为投资者的核心竞争力。而Python抓取股票技术,正是打通“数据获取—分析—决策”闭环的第一环。相较于传统手动采集方式,Python凭借其丰富的开源生态(如Requests、BeautifulSoup、Selenium、Pandas、Tushare、AKShare等),可实现自动化、高频率、低成本的金融数据采集。
本页面系统梳理了从基础原理到实战部署的全流程,涵盖以下核心价值:
据2024年量化行业调研显示,超过78%的个人量化投资者将Python抓取股票作为入门首选技能。其低门槛、高扩展性使其成为连接金融与编程的黄金桥梁。
不同数据源需匹配不同技术方案。以下为经过实战验证的组合:
环境初始化示例:
# 安装必要库
pip install requests beautifulsoup4 pandas selenium
pip install fake-useragent undetected-chromedriver
pip install tushare akshare sqlalchemy
特别注意:部分券商接口已启用设备指纹识别,需在Selenium中添加以下参数绕过检测:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from fake_useragent import UserAgent
options = Options()
options.add_argument('--disable-blink-features=AutomationControlled')
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
ua = UserAgent()
options.add_argument(f'user-agent={ua.random}')
driver = webdriver.Chrome(options=options)
driver.execute_cdp_cmd("Page.addScriptToEvaluateOnNewDocument", {
"source": """Object.defineProperty(navigator, 'webdriver', {get: () => undefined})"""
})
针对A股市场,推荐采用双数据源冗余策略。腾讯财经接口稳定但限制单次1000条,AKShare覆盖更全但需处理编码问题。
腾讯财经示例(沪深A股实时行情):
import requests
import pandas as pd
import time
def get_a_stock_realtime():
"""获取沪深A股实时行情(腾讯财经)"""
url = "http://qt.gtimg.cn/q=s_sh000001,sz399001,s_sh600000"
headers = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"
}
try:
resp = requests.get(url, headers=headers, timeout=10)
data = resp.text.strip().split(';')
stocks = []
for item in data:
if not item: continue
parts = item.split('~')
stocks.append({
'code': parts[2],
'name': parts[3],
'price': float(parts[31]),
'change': float(parts[32]),
'volume': int(parts[36]),
'amount': float(parts[37])
})
return pd.DataFrame(stocks)
except Exception as e:
print(f"获取失败: {e}")
return pd.DataFrame()
# 示例调用
df = get_a_stock_realtime()
print(df.head())
执行后将返回如下结构化数据:
| code | name | price | change | volume | amount |
|---|---|---|---|---|---|
| sh000001 | 上证指数 | 2987.34 | 1.25 | 245678 | 12.34 |
| sz399001 | 深证成指 | 9652.18 | 0.89 | 189012 | 8.76 |
AKShare补充方案(历史全量数据):
import akshare as ak
# 获取A股全部历史日线数据
stock_zh_a_daily = ak.stock_zh_a_daily(symbol="sh600000")
print(stock_zh_a_daily.head())
输出包含open/high/low/close/volume/amount等字段,支持自定义时间范围。建议配合pandas的resample方法进行周期转换。
港股市场数据源分散,推荐组合方案:雅虎财经API处理主流成分股,AKShare补充小盘股覆盖。
雅虎财经方案(需处理反爬):
import yfinance as yf
import pandas as pd
# 示例:获取腾讯控股(00700.HK)数据
ticker = "00700.HK"
stock = yf.Ticker(ticker)
hist = stock.history(period="1y")
print(hist.tail())
注意:雅虎财经对频繁请求会返回404,需添加重试机制:
import time
from tenacity import retry, stop_after_attempt, wait_exponential
@retry(stop=stop_after_attempt(3), wait=wait_exponential(multiplier=1, min=1, max=10))
def fetch_yahoo_data(symbol):
return yf.Ticker(symbol).history(period="1y")
# 调用
df = fetch_yahoo_data("00700.HK")
AKShare港股全量方案:
# 获取港股全部股票列表
hk_stock_list = ak.stock_hk_daily(symbol="00700")
print(hk_stock_list.head())
该方案覆盖超过3000只港股,数据延迟约15分钟,适合构建历史数据库。
美股市场数据开放度高,但需注册API Key。推荐IEX Cloud(免费层100万请求/月)和Alpha Vantage(免费层5次/分钟)。
IEX Cloud方案(需API Token):
import requests
import pandas as pd
API_TOKEN = "YOUR_IEX_CLOUD_TOKEN"
url = f"https://sandbox.iexapis.com/stable/stock/aapl/quote?token={API_TOKEN}"
response = requests.get(url)
data = response.json()
print(f"苹果公司最新价: ${data['iexRealtimePrice']}" if data['iexRealtimePrice'] else f"延迟价: ${data['close']}")
Alpha Vantage方案(免费版):
import requests
import time
API_KEY = "YOUR_ALPHA_VANTAGE_KEY"
url = f"https://www.alphavantage.co/query?function=TIME_SERIES_DAILY&symbol=IBM&apikey={API_KEY}"
response = requests.get(url)
time.sleep(12) # 免费版需间隔12秒
data = response.json()
print(data['Time Series (Daily)']['2024-03-20'])
重要提示:Alpha Vantage返回JSON格式需转换为DataFrame:
import json
df = pd.DataFrame(data['Time Series (Daily)']).T
df.index = pd.to_datetime(df.index)
df = df.astype(float)
print(df.head())
原始数据常存在以下问题,直接影响后续分析准确性:
标准清洗流程:
import pandas as pd
import numpy as np
def clean_stock_data(df):
"""股票数据清洗流水线"""
# 1. 去除重复行
df = df.drop_duplicates()
# 2. 处理缺失值:用前值填充(适用于非交易日)
df = df.fillna(method='ffill')
# 3. 价格异常检测(3σ原则)
for col in ['open', 'high', 'low', 'close']:
mean = df[col].mean()
std = df[col].std()
df = df[(df[col] > mean - 3*std) & (df[col] < mean + 3*std)]
# 4. 价格单位统一(部分数据以分为单位)
if df['close'].max() > 1000: # 假设超过1000元为分单位
df[['open','high','low','close']] = df[['open','high','low','close']] / 100
# 5. 添加关键衍生指标
df['price_change'] = df['close'].pct_change()
df['volatility'] = df['price_change'].rolling(5).std()
return df.sort_index()
# 示例调用
# df = pd.read_csv('raw_data.csv', parse_dates=['date'])
# clean_df = clean_stock_data(df)
# print(clean_df.describe())
特殊场景处理:
经清洗后的数据可直接用于特征工程,避免“垃圾进垃圾出”问题。
可视化是策略验证的关键环节。推荐使用Plotly实现交互式图表,兼顾美观与功能。
1. K线图(含MACD指标):
import plotly.graph_objects as go
from plotly.subplots import make_subplots
def plot_candlestick(df, symbol="AAPL"):
"""绘制K线图+MACD指标"""
fig = make_subplots(rows=2, cols=1,
shared_xaxes=True,
vertical_spacing=0.03,
row_heights=[0.7, 0.3])
# K线图
fig.add_trace(go.Candlestick(x=df.index,
open=df['open'],
high=df['high'],
low=df['low'],
close=df['close'],
name=symbol), row=1, col=1)
# MACD指标
df['ema12'] = df['close'].ewm(span=12).mean()
df['ema26'] = df['close'].ewm(span=26).mean()
df['macd'] = df['ema12'] - df['ema26']
df['signal'] = df['macd'].ewm(span=9).mean()
df['hist'] = df['macd'] - df['signal']
fig.add_trace(go.Bar(x=df.index, y=df['hist'],
marker_color=np.where(df['hist']>0, 'red', 'green'),
name='MACD Histogram'), row=2, col=1)
fig.add_trace(go.Scatter(x=df.index, y=df['macd'],
line=dict(color='blue', width=1), name='MACD'), row=2, col=1)
fig.add_trace(go.Scatter(x=df.index, y=df['signal'],
line=dict(color='orange', width=1), name='Signal'), row=2, col=1)
fig.update_layout(title=f'{symbol} K线与MACD指标',
yaxis_title='价格',
yaxis2_title='MACD',
xaxis_rangeslider_visible=False)
fig.show()
执行后可交互缩放查看历史点位,支持鼠标悬停显示具体数值。
2. 波动率热力图:
import seaborn as sns
import matplotlib.pyplot as plt
# 计算各股票20日波动率
volatility = df.groupby('code')['price_change'].std().unstack().T * np.sqrt(252)
plt.figure(figsize=(12, 8))
sns.heatmap(volatility, cmap='RdYlGn', center=0, annot=True, fmt=".2%")
plt.title('股票波动率热力图(年化)')
plt.show()
该图可快速识别高波动标的,辅助风险控制决策。
3. 行业资金流向:
import plotly.express as px
# 假设df含'industry'和'net_amount'字段
fig = px.bar(df.groupby('industry')['net_amount'].sum().sort_values(ascending=False).reset_index(),
x='industry', y='net_amount', color='net_amount',
color_continuous_scale='RdYlBu')
fig.update_layout(title='行业资金流向(亿元)')
fig.show()
实时监控主力资金动向,为行业轮动策略提供依据。
以5日/20日均线金叉/死叉为例,展示策略开发全流程:
import pandas as pd
import numpy as np
def double_ma_strategy(df, short_window=5, long_window=20):
"""双均线策略"""
df = df.copy()
df['short_ma'] = df['close'].rolling(short_window).mean()
df['long_ma'] = df['close'].rolling(long_window).mean()
# 信号生成
df['signal'] = 0
df.loc[df['short_ma'] > df['long_ma'], 'signal'] = 1 # 买入
df.loc[df['short_ma'] < df['long_ma'], 'signal'] = -1 # 卖出
# 交易逻辑
df['position'] = df['signal'].shift(1)
df['return'] = df['close'].pct_change()
df['strategy_return'] = df['position'] * df['return']
# 累计收益曲线
df['cum_return'] = (1 + df['return']).cumprod()
df['cum_strategy'] = (1 + df['strategy_return']).cumprod()
return df
# 示例调用
# df = pd.read_csv('600000.csv', parse_dates=['date'], index_col='date')
# result = double_ma_strategy(df)
# print(f"策略年化收益: {(result['cum_strategy'].iloc[-1]**252 - 1):.2%}")
关键优化点:
完整回测需使用Backtrader或Zipline等专业框架,此处仅展示核心逻辑。实际部署时需考虑:
实盘系统需满足:高可用性(7×24小时运行)、低延迟(毫秒级响应)、安全性(API密钥保护)。
基础部署方案:
关键代码示例:
import time
from apscheduler.schedulers.blocking import BlockingScheduler
from telegram import Bot
# 初始化Telegram机器人
bot = Bot(token="YOUR_TELEGRAM_BOT_TOKEN")
chat_id = "YOUR_CHAT_ID"
def execute_trade():
"""执行交易逻辑"""
try:
# 1. 获取最新信号
signal = generate_signal() # 自定义函数
# 2. 检查账户状态
if get_account_balance() < MIN_MARGIN:
bot.send_message(chat_id, "⚠️ 账户保证金不足!")
return
# 3. 执行下单(示例:华泰API)
if signal == 'buy':
order_id = send_order(symbol='600000', volume=100, price_type='FAK')
bot.send_message(chat_id, f"✅ 买入订单:{order_id}")
elif signal == 'sell':
order_id = send_order(symbol='600000', volume=100, price_type='FAK', side='sell')
bot.send_message(chat_id, f"❌ 卖出订单:{order_id}")
except Exception as e:
bot.send_message(chat_id, f"❌ 交易失败: {str(e)}")
# 定时任务(交易时段每分钟执行)
scheduler = BlockingScheduler()
scheduler.add_job(execute_trade, 'cron', hour='9-15', minute='*', timezone='Asia/Shanghai')
scheduler.start()
风险控制要点: