步骤/目录： 1.背景介绍 （1）MDC （2）Plex 2.MDC的使用 3.EverAver的使用 4.Plex的使用 （1）XBMCnfoMoviesImporter插件下载 （2）安装与设置 5.linux + docker安装plex （1）docker-compose的编写 （2）插件与设置 （3）头像 a.失败方法一 b.失败方法二 c.总结 d.方法三 e.方法四 f.修改nfo标签 （4）内网穿透 （5）显卡加速（未实操） a.卸载之前的CUDA、驱动 b.查看显卡是否可用CUDA c.安装NCT d.安装CRI-O e.配置 f.安装显卡驱动 g.安装驱动后检查 h.plex-docker开启硬件加速本文首发于个人博客https://lisper517.top/index.php/archives/92/，转载请注明出处。本文的目的是对本地电影文件进行整理，并搭建个人影音库。本文写作日期为2023年11月30日。用到的系统有win10、Ubuntu 22.04 LTS。1.背景介绍在 Python，爬虫与深度学习（17）——电影的信息爬取、下载与处理（二） 一文中，笔者曾经试图对本地电影文件进行压缩，并通过mysql库管理。压缩视频没有什么问题，但是根据视频文件名进行查询、归类时，由于文件名参差不齐，涉及的爬虫知识又比较复杂，笔者还想对视频内容进行比对，却一直没有好的解决办法。好在互联网上总是有人在做相似的项目，今天介绍的MDC就比较好地满足了这类需求。在文件管理上，笔者选择用废旧笔记本装linux连硬盘，Samba共享，详情参考 树莓派+Samba+frp搭建NAS（网络云存储） 。当然，树莓派的孱弱性能不太适合装plex，这里用笔记本代替。（1）MDCMovie_Data_Capture 刮削软件，项目地址 。可以识别本地视频文件名，到Java网站上查询，下载海报，生成nfo文件（这些工作称为刮削），根据主演名创建文件夹。其它类似的软件有AVDC，MDCx（原作者似乎在23-10-4删库），tinyMediaManager（正常情况下用的最多。这里不用tMM是因为本文主要讨论不正常的情况），EverAver（只能对单部电影刮削）等。另外，对于英语电影，可以使用tMM + 插件，具体可参考 提供插件的网站 （该网站目前转移到了 另一个地址 ）。在该插件网站注册一个账号，右上角点击头像可获得API token，MDCx可填入该API，或者tMM安装插件后也需填入该API。tMM插件的github项目地址 在此 ，把插件放到tMM安装路径的Addons文件夹里（新建一个）。（2）Plexplex官网 ，一个功能比较齐全的本地影音库解决方案。在存放电影文件的本地电脑上安装plex media server，只要该本地电脑能联网，就能在外网通过plex的客户端（安卓，苹果，Windows，Mac）或者网页端访问、管理电影文件。当然，在外网看是要花钱充会员的（在局域网内不用钱，所以也可以内网穿透）；且plex只能用自带的播放器。其实plex和流媒体有很多相似之处，也能看一些电视、美剧。需要注意的是，plex也可对电影进行识别（文件名，还会对视频内容进行比对），获得海报、生成nfo，但在一些细分领域的识别正确率不高。plex需要使用XBMCnfoMoviesImporter插件才能识别本地的nfo文件，这将在之后讲解。其它类似的有Jellyfin、Emby（Jellyfin是Emby的开源版本，Jellyfin疑似在文件较多时无法运行），Jvedio（可以用本地播放器，缺点是电影名的.必须改成-）等。plex是目前最成熟的影音管理软件，但是也有其它小问题。2.MDC的使用由于涉及爬虫，运行时请确认能够访问外网，最好是日服。可参考 Docker实践——如何查阅wiki百科及V2Ray 或者 Google云的免费使用。在 MDC发行版 中选择合适的版本进行下载。需要注意MDC最新的版本（7.0.1以后）开始内测，需要付费，有实力的读者可以支持一波。本文使用免费的6.6.7版本（最后一个免费版本）进行演示。对压缩文件解压，得到config.ini（配置文件）和exe文件。配置文件修改以下几处：stop_counter = 10 #默认为stop_counter = 0 rerun_delay = 1m #默认为rerun_delay = 0 [watermark] switch = 0 #默认为switch = 1 [storyline] switch = 0 #默认为switch = 1更多配置选项参考 配置说明 （该6.6.7版本配置说明网页在本文写作的这几天已失效）。接下来，将ini文件和exe文件放到存放电影的文件夹中（建议先用少量几部电影试验），把V2Ray改为全局模式，运行exe文件，等待一段时间。如果无法访问日服，MDC可能有些差错，可尝试上文提到的其它软件，推荐MDCx。第一次试刮削后，可自行更改配置，换其它刮削软件尝试，找到适于自己的软件及设置。3.EverAver的使用非常简单，把文件拖拽到软件框中即可。对于少量刮削出错的，使用EverAver单独刮削，或者使用其它软件的单独刮削功能。由于该软件19年后就没有更新过，现在也能用，但不推荐。4.Plex的使用（1）XBMCnfoMoviesImporter插件下载在github项目地址中按照最下面的指示操作即可，看不懂英文就看本文接下来的操作。首先下载XBMCnfoMoviesImporter的zip包，解压并将文件夹改名为 XBMCnfoMoviesImporter.bundle （此时该文件夹下应该有Contents文件夹，不要多套了一层），放到plex的插件目录下（Windows在安装目录的 Plex Media Server\Resources\Plug-ins-xxxxxxxxx 下，参考plex官网帮助）。如果除了电影，还有看电视剧的需求，可以装 XBMCnfoTVImporter插件 ，作者是同一个人。（2）安装与设置没什么特别的，安装即可。安装好后，右下角的任务栏会有plex图标，左键双击可点开网页，注册一个账号登录。在右上角 设置 - 管理 - 媒体库 -添加资料库 可新增电影库，在 高级 - 代理 中注意选择 XBMCnfoMoviesImporter ，plex才能识别本地的nfo文件。5.linux + docker安装plex在前文提到过，笔者使用的是 旧笔记本（Ubuntu 22.04）连硬盘+SMB分享 ，所以最好能在Ubuntu上装plex。步骤如下：（1）docker-compose的编写在Ubuntu命令行中如下操作：su root mkdir -p /docker/plex/conf/plex ls -l NAS硬盘挂载目录 chown -R 用户名:用户组 /docker/plex/conf id 用户名查看NAS硬盘目录所属的用户、用户组，修改目录到该用户及组下，记下uid、gid。vim /docker/plex/docker-compose.yml在docker-compose文件中写入如下内容：version: "3.9" services: plex: image: lscr.io/linuxserver/plex:latest container_name: plex network_mode: host environment: - PUID=1000 - PGID=1000 - TZ=Asia/Shanghai - VERSION=docker - PLEX_CLAIM= #optional volumes: - /docker/plex/conf/plex:/config - /path/to/tvseries:/tv - /path/to/movies:/movies restart: unless-stopped修改 PUID 和 PGID ；修改 /path/to/tvseries 和 /path/to/movies 分别为存放电视剧、电影的地方，或者随便怎么挂载，这个不是死的。需要注意的是 /config 是存放影音库等配置的地方，影像较多时留50GB的空间比较稳妥（等下也会将XBMCnfoMoviesImporter插件放到这里）。最后需要注意的是， /config 如果挂载到宿主机的机械硬盘上，plex会自动对视频截图作为封面，而不使用MDC刮削的封面，这可能与硬盘读取速度有关，所以 /config 一定要放在固态硬盘上。最后docker run：cd /docker/plex docker-compose config docker-compose up等待pull并成功run后，退出并后台运行：docker-compose up -d如果设置了防火墙，记得打开32400端口：ufw allow 32400（2）插件与设置插件方面，linux与windows下的插件是通用的。按前文的步骤，把XBMCnfoMoviesImporter.bundle放到 /docker/plex/conf/plex/Library/Application Support/Plex Media Server/Plug-ins 下即可，然后 docker-compose restart 一下。浏览器访问 笔记本本地ip:32400/web 即可进入plex media server网页。登录账号，跟随引导即可，添加电影库的操作也是一样的（如果在其它电脑上也装过plex，此时注意选择笔记本。此前实验时创建的其它电脑，在设置里可以删除），同样先用一个test文件夹、少量几部电影测试一下。笔者还进行了如下设置：允许我在家庭外访问我的媒体 取消勾选 启用电影预告片 取消勾选 启用视频预览缩略图 取消勾选 包含相关的额外内容 取消勾选对XBMCnfoMoviesImporter：Enable debug logging 勾选上（3）头像使用刮削软件生成的nfo文件，其中演员头像也是可以在plex中显示的（比如MDCx生成的nfo，演员头像在 <actor> 标签的 <thumb> 里，是 .actors/演员名.jpg ），但是需要一些操作。先到 gfriends项目 下载别人整理好的演员头像，然后进行一些设置。a.失败方法一根据这个提问下提问者的说法，在XBMC插件的设置中，把 actor thumb location 设置为link， path to movie library or global actor folder 不用管，在nfo里把thumb标签改成头像图片的绝对路径。失败，演员头像显示为黑，上面没有演员名的首字。b.失败方法二该提问者在后面的回答中提出，把 actor thumb location 设置为global， path to movie library or global actor folder 填存放演员头像的目录（如 /movies/actor_photo ，该目录下存放所有的头像jpg图片）。同样失败，同样显示黑图。以上两种方法，提问者都是在windows下运行，笔者在linux，可能有影响。c.总结在另一个问题中提到，存放头像的整个路径上都不能有空格，要用 _ 替代空格。源文件的 __init__.py 负责该设置，从750行的 if athumbloc in ['local','global']: 开始，这里其实有把空格替换成下划线。plex获得演员头像困难的根源在于它虽然管理本地电影，但核心是围绕线上刮削+本地缓存的，也就是说电影的海报、演员头像都需要从plex提供的数据库获取，缓存到自己的server上，完全没有考虑过server本地提供nfo、海报（不装XBMC插件，根本识别不了本地nfo），也不想让你碰缓存的数据。另一个问题是它会优先用本地缓存的图片，如果你更新了一些图片则它还是会显示以前的图片，解决方法可能是 扫描资料库文件 ，或者删除影音库再重建。在第一个提问中，提问者说头像显示为黑图说明plex已经缓存了图片（显示名字首字母说明没有缓存）。笔者在 /docker/plex/conf/plex 下查找了所有jpg图片，并没有缓存的头像。使用 grep -iR "actor photo" /docker/plex/conf/plex 也没发现有用的log信息（但是在最终的方法中还是找到了有用的log）。有人还提到，nfo里的thumb标签会覆盖掉 actor thumb location 和 path to movie library or global actor folder 设置，所以设置成local或者global时要删掉（这个不完全对，笔者设置link+ http://localhost 、thumb标签是 actor/演员名.jpg ，log里发现XBMC把它们拼接成了 http://localhostactor/演员名.jpg ）。d.方法三有人对XBMC插件进行了修改，名为 JAVnfoMoviesImporter.bundle ，安装方式和原来相同，不同在于它会在线从前面提到的gfriends项目获得演员头像。由于从github上刮削，可能需要翻越。而且gfriends项目作者已经停止更新，一些新人可能没有头像（这个可能性不大，因为还是有人在零散提交）。笔者没有使用这种方法，主要是想在本地存头像，一些演员的头像拍丑了、想用自己的图。e.方法四在第二个问题的最后，有人提出了另一种解决方案，稍微修改一下 __init__.py ，然后在docker-compose里另建一个nginx容器来提供头像链接；或者也可以不改 __init__.py ，只建nginx容器，笔者尝试也可以。不过改不改py文件，nfo文件都是要修改的。操作如下：cd /docker/plex docker-compose stop mkdir -p /docker/plex/conf/nginx chown -R 用户名:用户组 /docker/plex/conf vim /docker/plex/conf/nginx/default.conf写入nginx配置：server { listen 80; server_name localhost; location / { root /var/www/html; index index.php index.html index.htm; } }然后修改docker-compose.yml：vim /docker/plex/docker-compose.yml修改如下：version: "3.9" services: plex: image: lscr.io/linuxserver/plex:latest container_name: plex network_mode: host environment: - PUID=1000 - PGID=1000 - TZ=Asia/Shanghai - VERSION=docker - PLEX_CLAIM= #optional volumes: - /docker/plex/conf/plex:/config - /path/to/tvseries:/tv - /path/to/movies:/movies restart: unless-stopped nginx: image: nginx:1.23.1 ports: - "80:80" volumes: - /docker/plex/conf/nginx:/etc/nginx/conf.d - 存放头像的目录:/var/www/html logging: driver: syslog restart: always如果你已经用了笔记本的80端口，也可以改成其它端口（比如改成9090端口： 9090:80 ）。ufw allow 80 cd /docker/plex docker-compose config docker-compose up -d现在测试一下nginx图片服务器能否正常工作。在浏览器打开 笔记本内网ip/图片名.jpg （或者 笔记本内网ip:9090/图片名.jpg ），应该能看到头像图片。最后设置一下plex（其实就是保持默认设置）， actor thumb location 设置为link， path to movie library or global actor folder 保持为 http://localhost （可以直接删掉并保存，XBMC会自动填充这个值）。重点是nfo文件里，thumb标签的值，改成 http://localhost/演员名.jpg 即可（如果nginx用的不是80端口，这里应该填 http://localhost:端口号/演员名.jpg ）（有些刮削软件填的是 .actors/演员名.jpg ，适用于EMBY等其它影音管理软件）。这时nfo里的thumb标签确实会覆盖掉头像的链接。测试一下，把之前测试的几部电影剪切走，换成其它新的电影，能正常显示演员头像。最后修改一下nginx的设置，不要把nginx容器的端口暴露到宿主机上。如下操作：ufw delete allow 80 cd /docker/plex docker-compose stop vim /docker/plex/docker-compose.yml把端口映射删掉，如下：version: "3.9" services: plex: image: lscr.io/linuxserver/plex:latest container_name: plex network_mode: host environment: - PUID=1000 - PGID=1000 - TZ=Asia/Shanghai - VERSION=docker - PLEX_CLAIM= #optional volumes: - /docker/plex/conf/plex:/config - /path/to/tvseries:/tv - /path/to/movies:/movies restart: unless-stopped nginx: image: nginx:1.23.1 volumes: - /docker/plex/conf/nginx:/etc/nginx/conf.d - 存放头像的目录:/var/www/html logging: driver: syslog restart: always然后删除并重新构建：docker-compose stop nginx docker-compose rm nginx #按y确认 docker-compose config docker-compose up -d需要注意的是thumb标签的值稍微有所不同。docker container ls其中会有一个 plex-nginx-1 容器（老版本的docker-compose可能是plex_nginx_1或者其它名称）。查看它在容器组合里的ip：docker inspect plex-nginx-1 | grep IPAddress比如说是172.22.0.2，那么thumb标签的值就改成 http://172.22.0.2/演员名.jpg 。如果对docker有经验的读者，可能会想改成 http://nginx/演员名.jpg 或者 http://plex-nginx-1/演员名.jpg ，可以进入plex容器来验证：docker exec -it plex bash cd /tmp wget http://nginx/演员名.jpg wget http://plex-nginx-1/演员名.jpg wget http://172.22.0.2/演员名.jpgplex容器自带了wget命令，但最终只有改成ip值才行。f.修改nfo标签把所有thumb标签的值从 .actors/演员名.jpg 改成 http://172.22.0.2/演员名.jpg 。使用如下python脚本：#!/usr/bin/python3 # -*- coding: UTF-8 -*- import os dir = r'电影存放目录' content = '' def main(): global content for dirpath, dirnames, filenames in os.walk(dir): for filename in filenames: suffix = os.path.splitext(filename)[-1].upper() if suffix != '.NFO': continue abs_path = os.path.join(dir, dirpath, filename) with open(abs_path, 'r+', encoding='utf-8') as f: content = f.read() if content.find(r'<thumb>.actors/') == -1 and content.find( r'<thumb>http://172.22.0.2/') == -1: print(f'没有头像{abs_path}') content = content.replace(r'<thumb>.actors/', r'<thumb>http://172.22.0.2/') f.close() with open(abs_path, 'w', encoding='utf-8') as f: f.write(content) f.close() if __name__ == "__main__": main() 每次有新文件，刮削后运行一次即可。（4）内网穿透plex在外网访问需要开会员，2023年，费用大约是包月$5、包月$40、永久$120，有需要的可以购买。从学术研究的角度，也可以尝试用内网穿透（前提是有公网ip，像前文提到的Google云服务器），比如用frp，映射笔记本的32400端口即可。可参考 Docker树莓派实践——frp（附dashboard开启https、frp双向验证） ；或者能访问ipv6的，笔者更建议 IPv6 + DDNS （能跑满家里的上传带宽），见 IPv6与DDNS ，这里不再赘述。（5）显卡加速（未实操）如果在自己的电脑上观看该电脑上的视频，会使用该电脑的显卡解码；如果用自己的电脑观看云视频，就涉及到在服务器端还是在客户端解码的问题。plex解码笔者不太清楚，按常理来说用手机看最好在服务器端解码（本文为例就是笔记本上解码），用PC就在客户端解码。如果笔记本用的是Nvidia某些型号的显卡，可以用显卡解码，毕竟CPU硬解码还是比较占CPU。笔者测试时，观看720p电影，CPU占用不大（ top 命令，或者 watch -d -n 1 cat /usr/local/watchdog/cpu_temperature 看CPU温度），没有开硬件加速的必要。在 docker-plex 的 Nvidia 小节，提到了开启显卡加速的方法，即先安装 nvidia-docker ，然后运行一个nvidia容器（加 --runtime=nvidia ），plex容器则在yml里写 -e NVIDIA_VISIBLE_DEVICES=all （或者把 all 换成UUID来指定特定显卡。 nvidia-smi --query-gpu=gpu_name,gpu_uuid --format=csv 可查看GPU的UUID）。可惜的是，截止23年12月，nvidia-docker这个项目已经废弃了，转而被NVIDIA Container Toolkit取代。接下来就演示用NCT为plex开启显卡加速（笔者没有实操）。a.卸载之前的CUDA、驱动如果之前装过CUDA，或者在本次流程装CUDA时出现了意外，可以卸载后重装。根据 Nvidia官方文档 ，执行以下命令：sudo apt-get --purge remove "*cuda*" "*cublas*" "*cufft*" "*cufile*" "*curand*" \ "*cusolver*" "*cusparse*" "*gds-tools*" "*npp*" "*nvjpeg*" "nsight*" "*nvvm*" sudo apt-get --purge remove "*nvidia*" "libxnvctrl*" sudo apt-get autoremoveb.查看显卡是否可用CUDAupdate-pciids lspci | grep -i nvidia获取当前的显卡型号，到 NVIDIA官网 比对，是否能开启CUDA。如果不能，就不需要进行接下来的步骤了（没有对应的显卡驱动）。根据官方的 这个回答 ，2008年以后制造的英伟达显卡都支持CUDA（包括笔记本），所以N卡其实可以无脑装。c.安装NCT参考 NCT安装官方指南 。为笔记本开启翻越，进行如下操作（Ubuntu下。其它发行版参考原文）：curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \ && curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \ sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \ sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list可以更新一些实验性功能包，一般情况下没必要。sed -i -e '/experimental/ s/^#//g' /etc/apt/sources.list.d/nvidia-container-toolkit.list安装即可。apt-get update apt-get install -y nvidia-container-toolkitd.安装CRI-O接下来需要用到CRI-O，不太清楚是什么东西，好像是docker的某种替代或者补充，笔者还没装过。安装CRI-O（参考 这篇文章 ）：apt update apt install apt-transport-https ca-certificates curl gnupg2 software-properties-common -y export OS=xUbuntu_22.04 export CRIO_VERSION=1.24 echo "deb https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/$OS/ /"| sudo tee /etc/apt/sources.list.d/devel:kubic:libcontainers:stable.list echo "deb http://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable:/cri-o:/$CRIO_VERSION/$OS/ /"|sudo tee /etc/apt/sources.list.d/devel:kubic:libcontainers:stable:cri-o:$CRIO_VERSION.list curl -L https://download.opensuse.org/repositories/devel:kubic:libcontainers:stable:cri-o:$CRIO_VERSION/$OS/Release.key | sudo apt-key add - curl -L https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/$OS/Release.key | sudo apt-key add - apt update apt install cri-o cri-o-runc -y systemctl start crio systemctl enable crio systemctl status crio参考文章后面还有一些附加的内容，这里没有继续。e.配置第一个配置会调整宿主机的 /etc/docker/daemon.json 文件，使得docker可以使用NVIDIA Container Runtime。nvidia-ctk runtime configure --runtime=docker systemctl restart docker第二个配置会调整 /etc/containerd/config.toml 文件，使得containerd可以使用NVIDIA Container Runtime。nvidia-ctk runtime configure --runtime=containerd systemctl restart containerd第三个配置调整 /etc/crio/crio.conf 文件，使得CRI-O可以使用NVIDIA Container Runtime。nvidia-ctk runtime configure --runtime=crio systemctl restart criof.安装显卡驱动参考 官网文章 。apt-get install linux-headers-$(uname -r) distribution=$(. /etc/os-release;echo $ID$VERSION_ID | sed -e 's/\.//g') wget https://developer.download.nvidia.com/compute/cuda/repos/$distribution/x86_64/cuda-keyring_1.0-1_all.deb dpkg -i cuda-keyring_1.0-1_all.deb apt-get update apt-get -y install cuda-drivers这个是通过包管理安装，没有 /usr/local/cuda-版本号/nvcc 。g.安装驱动后检查参考 官方说明 。h.plex-docker开启硬件加速运行一个CUDA容器（这里笔者不是很确定）：docker run --runtime=nvidia --gpus all ubuntu nvidia-smi在plex-docker的docker-compose.yml中增加环境参数： -e NVIDIA_VISIBLE_DEVICES=all ，重新build即可。
Python，爬虫与深度学习（16）——电影的信息爬取、下载与处理（一）
步骤/目录： 1.需求介绍 2.准备工作 （0）安装python库 （1）创建scrapy项目 （2）创建并运行mysql （3）创建并运行redis （4）创建python自建库 a.log.py b.redis.py c.mysql.py d.interaction.py e.torrent_parser.py f.其他 3.scrapy项目的修改 （1）items.py （2）middlewares.py （3）pipelines.py （4）settings.py （5）爬虫 4.使用说明本文首发于个人博客https://lisper517.top/index.php/archives/55/，转载请注明出处。本文的目的是使用scrapy爬取电影的信息。本文写作日期为2022年10月3日。运行的平台为win10和树莓派，编辑器为VS Code。1.需求介绍在日常生活中，我们常常会想要看一些精彩的电影，但是看电影又比较费时间，所以最好能把电影信息爬取下来，通过一些方法筛选出想要的电影，再通过一些标准筛选出对应的磁力链接或其他下载链接，下载电影到本地，最后对本地存储的电影信息进行入库、对占用空间较大的电影进行压缩等。本文是系列文章的第一篇，目的是使用scrapy从电影信息网站上爬取数据。为了方便，本文选择了一个能同时提供电影磁链的网站，该网站提供了演员主页，收录了每位演员出演的电影。由于笔者挑选电影时有演员偏好，所以打算收藏喜欢的演员的主页，根据演员来挑选电影。2.准备工作（0）安装python库笔者也搞不太清楚需要哪些库了，不过推荐安装的有以下这些：Scrapy（python-scrapy）、requests（处理网络请求）、bs4（对响应内容进行解析）、pymysql（连接mysql数据库）、lxml（网页解析器）、cryptography（连接mysql时密码加密）、selenium（操控浏览器）、redis（python-redis）。如果后续运行时有些依赖的库没安装，请自行安装。（1）创建scrapy项目在爬取磁链时，这些数据是通过javascript动态生成的，所以这个项目就命名为javaspider。在win10电脑上的cmd中运行如下命令：D: cd D:\spiders\scrapy scrapy startproject javaspider scrapy genspider java（2）创建并运行mysql这里选择在树莓派上通过docker运行mysql，具体的操作步骤见 Python，爬虫与深度学习（10）——scrapy+mysql构建ipool 这篇文章。mysql中主要有两张表，movie和torrent，分别存储电影和磁链的信息。库、表的创建将使用python，这里不单独创建。（3）创建并运行redis这里还要使用到redis，因为一个演员不止出演一部电影，一部电影通常也不止一个tag，并且筛选磁链时还可能用到磁链中的文件（也是一对多的关系）。这种一对多、无重复的表用redis的集合比较容易实现，而用mysql这样的关系型数据库比较复杂。由于redis对内存的要求比较大，这里就在win10机器上运行docker-redis。在windows上安装docker（即安装docker-desktop）的方法见 Docker入门（一）。在win10上运行容器的过程是一样的，比如cmd执行如下命令：D: md D:\docker\redis\data\redis然后新建一个 D:\docker\redis\docker-compose.yml compose模板文件，写入以下内容：version: "3.9" services: redis: image: redis ports: - "56379:6379" volumes: - D:\docker\redis\data\redis:/data logging: driver: json-file restart: always 最后运行：cd D:\docker\redis docker-compose config docker-compose up成功运行redis。注意这种方法运行的redis是完全没有安全措施的，不要开放到外网了。另外，为了使用redis-cli，可以在wsl环境（如果你是按照 Docker入门（一） 的方法在win10上运行docker，应该清楚wsl是什么）里安装redis。在cmd中进行如下操作：wsl -d Ubuntu-20.04 #如果你也用的是Ubuntu-20.04的话 cd /mnt/d/docker/redis wget https://download.redis.io/releases/redis-7.0.4.tar.gz #最新版本的redis tar -xzf redis-7.0.4.tar.gz cd redis-7.0.4 apt-get install pkg-config make gcc libssl-dev #一些依赖项 make BUILD_TLS=yes安装完成后，用这里的redis-cli连接本地的redis：cd /mnt/d/docker/redis/redis-7.0.4/src ./redis-cli -p 56379 ping #redis回应 PONG至此windows上redis、redis-cli安装完成。（4）创建python自建库创建python自建库的方法见 Python，爬虫与深度学习（7）——建立自己的库 。这里需要使用的自建库主要有以下几个文件：a.log.py这个主要是获取一个logger对象，如下：#!/usr/bin/python # -*- coding: UTF-8 -*- import logging, time, os def get_logger(abs_path: str, log_file_name: str = '', getLogger_name: str = ''): '''传入绝对路径和log文件名，返回log对象。 :param abs_path: log文件存储的绝对路径 :param log_file_name: log文件名，若log文件名为空，则使用默认的文件名 :param getLogger_name: logger对象名，如果在同一个程序中需要同时写多个log文件，建议这个参数不要相同 :return log对象''' try: formatter = logging.Formatter( '%(lineno)d : %(asctime)s : %(levelname)s : %(funcName)s : %(message)s' ) if log_file_name == '': log_file_name = 'log#{}.txt'.format(time.strftime('-%Y-%m-%d')) fileHandler = logging.FileHandler( (os.path.join(abs_path, log_file_name)), mode='w', encoding='utf-8') fileHandler.setFormatter(formatter) if getLogger_name == '': getLogger_name = 'logger' log = logging.getLogger(getLogger_name) log.setLevel(logging.DEBUG) log.addHandler(fileHandler) return log except: raise Exception('error: MyPythonLib.log.get_logger failed ')b.redis.py返回redis的连接池：#!/usr/bin/python # -*- coding: UTF-8 -*- import redis def get_pool(database_num: int = 0, decode_responses=True): '''指定数据库编号，返回redis连接池。 :param database_num: 数据库编号，不指定时默认用0号 :param decode_responses: 是否自动解析数据。如果只存string，设置为True；如果要存二进制对象，设置为False :return redis.ConnectionPool''' try: pool = redis.ConnectionPool( host='192.168.1.2', port=56379, db=database_num, decode_responses=decode_responses, ) return pool except: raise Exception('MyPythonLib.redis.get_pool failed ')记得把 redis.ConnectionPool 方法里的参数改成自己的。c.mysql.py主要是连接mysql、字符串转换。#!/usr/bin/python # -*- coding: UTF-8 -*- import pymysql, os, logging, time def get_conn(database_name: str = 'mysql'): '''指定数据库名，返回mysql连接对象。 :param database_name: 数据库名，不指定时默认用mysql库 :return Connection''' try: conn = pymysql.connect(host='192.168.1.1', port=53306, user='root', passwd='mypasswd', db=database_name, charset='utf8') return conn except: raise Exception('MyPythonLib.mysql.get_conn failed ') def str_into_mysql(string: str) -> str: '''输入一个字符串，将其转换为可以在mysql中储存的形式。 这个字符串的使用位置是列值处，比如 "SELECT * FROM 表名 WHERE 列名='字符串';" 中的字符串，若包含 \\ 、 ' 、 " 这三个字符，就需要用本函数处理。 :param string: 目标字符串 :return 可以存入mysql的字符串 ''' try: string = string.replace('\\', '\\\\') string = string.replace("'", "\\'") string = string.replace('"', '\\"') return string except: raise Exception('MyPythonLib.mysql.str_into_mysql failed ') 记得把 pymysql.connect 方法里的参数改成自己的。d.interaction.py这个文件目前只需要用到一个显示进度的函数，内容如下：#!/usr/bin/python # -*- coding: UTF-8 -*- import time, os def print_progress(start_time: float, progress: int, total: int, msg: str, cls=False): '''图形化打印进度及附加信息（清空屏幕）。 :param start_time: 开始时间，由time.time()得到的float :param progress: 整数，表示目前进度 :param total: 整数，表示总量 :param msg: 附加信息 :param cls: 是否清空屏幕（windows下）''' try: percent = 100. * progress / total x = int(2 * percent / 5) duration = time.time() - start_time h, m, s = duration // 3600, duration // 60 % 60, duration % 60 time_format = '{:0>2.0f}:{:0>2.0f}:{:0>2.0f}'.format(h, m, s) print('\r已运行' + time_format + ' [' + '#' * x + '.' * (40 - x) + ']', '{:.2f}%'.format(percent), str(progress) + '/' + str(total), msg, end='') if cls: os.system('cls') except: raise Exception( 'error: MyPythonLib.interaction.print_progress failed ') e.torrent_parser.py用于解析torrent文件，主要用于根据torrent文件得到磁链。#!/usr/bin/python # -*- coding: UTF-8 -*- import collections, hashlib def bencode(elem): if type(elem) == str: elem = str.encode(elem) if type(elem) == bytes: result = str.encode(str(len(elem))) + b":" + elem elif type(elem) == int: result = str.encode("i" + str(elem) + "e") elif type(elem) == list: result = b"l" for item in elem: result += bencode(item) result += b"e" elif type(elem) in [dict, collections.OrderedDict]: result = b"d" for key in elem: result += bencode(key) + bencode(elem[key]) result += b"e" return result def bdecode(bytestr, recursiveCall=False): startingChars = dict({b"i": int, b":": str, b"l": list, b"d": dict}) digits = [b"0", b"1", b"2", b"3", b"4", b"5", b"6", b"7", b"8", b"9"] started = ended = False curtype = None numstring = b"" # for str, int result = None # for list, dict key = None # for dict while len(bytestr) > 0: # reading and popping from the beginning char = bytestr[:1] if not started: bytestr = bytestr[1:] if char in digits: numstring += char elif char in startingChars: started = True curtype = startingChars[char] if curtype == str: size = int(bytes.decode(numstring)) # try to decode strings try: result = bytes.decode(bytestr[:size]) except UnicodeDecodeError: result = bytestr[:size] bytestr = bytestr[size:] ended = True break elif curtype == list: result = [] elif curtype == dict: result = collections.OrderedDict() else: raise ValueError("Expected starting char, got ‘" + bytes.decode(char) + "’") else: # if started if not char == b"e": if curtype == int: bytestr = bytestr[1:] numstring += char elif curtype == list: item, bytestr = bdecode(bytestr, recursiveCall=True) result.append(item) elif curtype == dict: if key == None: key, bytestr = bdecode(bytestr, recursiveCall=True) else: result[key], bytestr = bdecode(bytestr, recursiveCall=True) key = None else: # ending: char == b"e" bytestr = bytestr[1:] if curtype == int: result = int(bytes.decode(numstring)) ended = True break if ended: if recursiveCall: return result, bytestr else: return result else: raise ValueError("String ended unexpectedly") class torrent(): '''解析种子文件。''' def __init__(self, abs_torrent_file_path: str) -> None: ''':param abs_torrent_file_path: 种子文件的绝对路径''' self.abs_torrent_file_path = abs_torrent_file_path bytes_stream = '' with open(abs_torrent_file_path, 'rb') as f: bytes_stream = f.read() self.metadata = bdecode(bytes_stream) encodedInfo = bencode(self.metadata["info"]) self.hash_value = hashlib.sha1(encodedInfo).hexdigest().upper() if __name__ == '__main__': pass参考了github上的 这个项目 。不得不说稍微有点深度的问题还是github靠谱，csdn纯纯的互相乱抄。f.其他在之前的 Python，爬虫与深度学习（10）——scrapy+mysql构建ipool 一文中，笔者把mysql的api写到了自建库的mysql.py里，现在感觉这样不太合适，写到scrapy项目里更好一点，毕竟自建库应该写一些更基础、通用的功能。于是新建一个libs目录： D:\spiders\scrapy\javaspider\javaspider\libs ，里面放一个空的 __init__.py 文件，和一个 java_extensions.py ，它主要是为scrapy项目提供一些功能，但这些功能又不适合放在自建库里，比如mysql、redis的api。内容如下：#!/usr/bin/python # -*- coding: UTF-8 -*- import os, logging, time, redis, re, datetime from MyPythonLib import log as mylog from MyPythonLib import redis as myredis from MyPythonLib.torrent_parser import torrent import urllib.parse ORIGIN_URL_JAVA = r'https://www.java.com' #电影网站的url TEMP_URL_JAVA = r'https://www.seejava.com' #有时候该电影网站无法访问，需要换成临时网址 TORRENT_DIR = r'D:\torrents' #存放下载过的磁链的目录 def get_actor_main_pages() -> list[str]: '''收藏演员主页（或者搜索页，或者根页面）->复制网址到txt文件->根据相对url和TEMP_URL_JAVA提供待爬url :return 待爬的url列表''' with open(os.path.join(os.getcwd(), 'actor_main_pages.txt'), 'r', encoding='utf-8') as f: abs_urls = f.read().split('\n') rel_urls_set = set() for abs_url in abs_urls: if re.search(r'\w', abs_url) != None: base_url = re.search( r'((https://)|(http://)){0,1}(\w+\.){0,1}(\w+\.){1}\w+', abs_url).group() rel_url = abs_url.replace(base_url, '') if rel_url in rel_urls_set: print('重复网址：{}'.format(rel_url)) rel_urls_set.add(rel_url) abs_urls = [] for rel_url in rel_urls_set: abs_url = urllib.parse.urljoin(TEMP_URL_JAVA, rel_url) abs_urls.append(abs_url) return abs_urls class database_api_java(): """通过这个接口实现提交爬取到的作品信息。还可以更新数据、取出磁链。""" conn = mysql.get_conn() m_cur = conn.cursor() #这个cursor是类公用的，用于不需要实例化对象的mysql操作 pool = myredis.get_pool(database_num=0) r_cur = redis.Redis( connection_pool=pool) #这个cursor是类公用的，用于不需要实例化对象的redis操作 def __init__(self, log: logging.Logger = None): """:param log: 可选的log对象，若不提供，将在当前文件夹下新建logs目录、创建log文件""" if log == None: log_dir = os.path.join(os.getcwd(), 'logs') if not os.path.isdir(log_dir): os.mkdir(log_dir) log_file_name = 'database_api_java#{}.txt'.format( time.strftime('-%Y-%m-%d_%H_%M_%S')) log = mylog.get_logger(log_dir, log_file_name, 'mysql_api_java') self.log = log message = 'database_api_java object initialized' self.log.info(message) self.m_cur = database_api_java.conn.cursor() #对象自己的cur self.r_cur = redis.Redis(connection_pool=database_api_java.pool) database_api_java.check_table() message = 'database java and tables checked' self.log.info(message) def __del__(self): self.m_cur.close() self.r_cur.close() message = 'mysql_api_java object deleted' self.log.info(message) @staticmethod def check_table(): """检查一下mysql中是否已经建好了表，若表不存在则新建表。 :这里并没有检查：若表存在，表的格式是否正确。 mysql中movie表各列含义与默认值： :no，自增主键； :code，作品识别码（"unknown"）； :full_name，作品全名（"unknown"）； :release_date，发行日期（"1000-01-01"）； :duration，持续时间，单位为分钟（0）； :director，导演名（"unknown"）； :producer，制作商（"unknown"）； :publisher，发行商（"unknown"）； :series，系列（"unknown"）； :url，来源网页（"unknown"）； :downloaded，是否下载（0）； :file_path，如果已下载，文件位置（"unknown"）； :created，该行创建时间。 mysql中torrent表各列含义与默认值： :no，自增主键； :hash_value，hash值（"unknown"）； :movie_no，对应电影编号（0）； :name，名称（"unknown"）； :file_num，包含文件数量（0）； :size，文件大小（单位为MB，0）； :high_definition，是否高清（0）； :record_date，网站收录日期（"1000-01-01"）。 redis中actor_movie： 集合，键为 java:actor:名字 ，值为电影在mysql中的编号。 比较特殊的是 java:actor:unknown ，这个键里存的是演员未知的电影。 redis中tag_movie： 集合，键为 java:tag:标签名 ，值为电影在mysql中的编号。 比较特殊的是 java:tag:no ，这个键里存的是没有tag的电影。 redis中torrent_files： 集合，键为 java:torrent:编号 ，值为文件名。""" conn, m_cur = database_api_java.conn, database_api_java.m_cur command_movie = ''' CREATE TABLE IF NOT EXISTS movie ( no BIGINT UNSIGNED AUTO_INCREMENT, code VARCHAR(30) NOT NULL DEFAULT "unknown" COMMENT "作品识别码", full_name VARCHAR(400) NOT NULL DEFAULT "unknown" COMMENT "作品全名", release_date DATE NOT NULL DEFAULT "1000-01-01" COMMENT "发行日期", duration SMALLINT UNSIGNED NOT NULL DEFAULT 0 COMMENT "持续时间（min）", director VARCHAR(200) NOT NULL DEFAULT "unknown" COMMENT "导演名", producer VARCHAR(200) NOT NULL DEFAULT "unknown" COMMENT "制作商", publisher VARCHAR(200) NOT NULL DEFAULT "unknown" COMMENT "发行商", series VARCHAR(200) NOT NULL DEFAULT "unknown" COMMENT "系列", url VARCHAR(200) NOT NULL DEFAULT "unknown" COMMENT "来源网页", downloaded BIT(1) NOT NULL DEFAULT 0 COMMENT "是否下载", file_path VARCHAR(500) NOT NULL DEFAULT "unknown" COMMENT "如果已下载，文件位置", created TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP, PRIMARY KEY (no));''' command_movie = command_movie.replace('\n', '') command_movie = command_movie.replace(' ', '') command_torrent = ''' CREATE TABLE IF NOT EXISTS torrent ( no BIGINT UNSIGNED AUTO_INCREMENT, hash_value VARCHAR(200) NOT NULL DEFAULT "unknown" COMMENT "hash值", movie_no BIGINT UNSIGNED NOT NULL DEFAULT 0 COMMENT "电影编号", name VARCHAR(400) NOT NULL DEFAULT "unknown" COMMENT "名称", file_num INT UNSIGNED NOT NULL DEFAULT 0 COMMENT "包含文件数量", size INT UNSIGNED NOT NULL DEFAULT 0 COMMENT "文件大小（MB）", high_definition BIT(1) NOT NULL DEFAULT 0 COMMENT "是否高清", record_date DATE NOT NULL DEFAULT "1000-01-01" COMMENT "网站收录日期", PRIMARY KEY (no));''' command_torrent = command_torrent.replace('\n', '') command_torrent = command_torrent.replace(' ', '') try: m_cur.execute("CREATE DATABASE IF NOT EXISTS java;") m_cur.execute("USE java;") m_cur.execute(command_movie) m_cur.execute(command_torrent) conn.commit() except: conn.rollback() raise Exception('java_extensions.database_api_java failed') def save_movie(self, movie_dict: dict): """存入一个movie（以字典形式提交）。注意这里只是对不存在的数据添加默认值，没有验证数据类型等。 工作流程：向mysql的movie表中存入数据；向redis的tag_movie存入数据；向redis的actor_movie存入数据。 :param movie_dict: 包含movie数据的字典。 :return movie_no: 电影的编号""" conn, m_cur, r_cur, log = database_api_java.conn, self.m_cur, self.r_cur, self.log if len(movie_dict) == 0: message = 'no movie data submitted' log.info(message) return command_movie = '''INSERT INTO movie (code,full_name,release_date,duration,director,producer,publisher,series,url,downloaded,file_path) VALUES ''' code, full_name, release_date, duration, director, producer, publisher, series, url, downloaded, file_path = 'unknown', 'unknown', "1000-01-01", 0, 'unknown', 'unknown', 'unknown', 'unknown', 'unknown', 0, 'unknown' tags, actors = set(['no']), set(['unknown']) if 'code' in movie_dict.keys(): code = mysql.str_into_mysql(movie_dict['code']) if 'full_name' in movie_dict.keys(): full_name = mysql.str_into_mysql(movie_dict['full_name']) if 'release_date' in movie_dict.keys(): release_date = movie_dict['release_date'] if release_date == '0000-00-00': release_date = '1000-01-01' if 'duration' in movie_dict.keys(): duration = movie_dict['duration'] if 'director' in movie_dict.keys(): director = mysql.str_into_mysql(movie_dict['director']) if 'producer' in movie_dict.keys(): producer = mysql.str_into_mysql(movie_dict['producer']) if 'publisher' in movie_dict.keys(): publisher = mysql.str_into_mysql(movie_dict['publisher']) if 'series' in movie_dict.keys(): series = mysql.str_into_mysql(movie_dict['series']) if 'url' in movie_dict.keys(): url = mysql.str_into_mysql(movie_dict['url']) if 'downloaded' in movie_dict.keys(): downloaded = movie_dict['downloaded'] if 'file_path' in movie_dict.keys(): file_path = mysql.str_into_mysql(movie_dict['file_path']) if 'tags' in movie_dict.keys(): tags = movie_dict['tags'] if tags == None: tags = set(['no']) if 'actors' in movie_dict.keys(): actors = movie_dict['actors'] if actors == None: actors = set(['unknown']) command_movie += '("{}","{}","{}",{},"{}","{}","{}","{}","{}",{},"{}"),'.format( code, full_name, release_date, duration, director, producer, publisher, series, url, downloaded, file_path) command_movie = command_movie[:-1] + ';' try: m_cur.execute(command_movie) conn.commit() m_cur.execute( 'SELECT MAX(no) FROM movie WHERE code="{}" AND full_name="{}";' .format(code, full_name)) movie_no = str(m_cur.fetchall()[0][0]) pipe = r_cur.pipeline() for tag in tags: key = 'java:tag:{}'.format(tag) value = movie_no pipe.sadd(key, value) for actor in actors: key = 'java:actor:{}'.format(actor) value = movie_no pipe.sadd(key, value) pipe.execute() message = 'saved movie successfully, code:full_name={}:{}'.format( code, full_name) log.info(message) return movie_no except: conn.rollback() log.info('mysql错误命令为：' + command_movie) raise Exception('java_extensions.save_movie failed') def save_torrent(self, torrent_dict: dict): """存入一个torrent（以字典形式提交）。注意这里只是对不存在的数据添加默认值，没有验证数据类型等。 工作流程：向mysql的torrent表中存入数据；向redis的torrent_files存入数据。 :param torrent_dict: 包含torrent数据的字典。""" conn, m_cur, r_cur, log = database_api_java.conn, self.m_cur, self.r_cur, self.log if len(torrent_dict) == 0: message = 'no torrent data submitted' log.info(message) return command_torrent = '''INSERT INTO torrent (hash_value,movie_no,name,file_num,size,high_definition,record_date) VALUES ''' hash_value, movie_no, name, file_num, size, high_definition, record_date = 'unknown', 0, 'unknown', 0, 0, 0, "1000-01-01" torrent_files = set() if 'hash_value' in torrent_dict.keys(): hash_value = mysql.str_into_mysql(torrent_dict['hash_value']) if 'movie_no' in torrent_dict.keys(): movie_no = int(torrent_dict['movie_no']) if 'name' in torrent_dict.keys(): name = mysql.str_into_mysql(torrent_dict['name']) if 'file_num' in torrent_dict.keys(): file_num = torrent_dict['file_num'] if 'size' in torrent_dict.keys(): size = torrent_dict['size'] if size == '': size = 0 if 'high_definition' in torrent_dict.keys(): high_definition = torrent_dict['high_definition'] if 'record_date' in torrent_dict.keys(): record_date = torrent_dict['record_date'] if record_date == '0000-00-00': record_date = '1000-01-01' if 'torrent_files' in torrent_dict.keys(): torrent_files = torrent_dict['torrent_files'] command_torrent += '("{}",{},"{}",{},{},{},"{}"),'.format( hash_value, movie_no, name, file_num, size, high_definition, record_date) command_torrent = command_torrent[:-1] + ';' try: m_cur.execute(command_torrent) conn.commit() m_cur.execute( 'SELECT MIN(no) FROM torrent WHERE hash_value="{}";'.format( hash_value)) torrent_no = str(m_cur.fetchall()[0][0]) key = 'java:torrent:{}'.format(torrent_no) pipe = r_cur.pipeline() for torrent_file in torrent_files: value = torrent_file pipe.sadd(key, value) pipe.execute() message = 'saved torrent successfully, hash_value={}'.format( hash_value) log.info(message) except: conn.rollback() log.info('mysql错误命令为：' + command_torrent) raise Exception('java_extensions.save_torrent failed') def update_movie(self, movie_dict: dict): """更新一个movie（以字典形式提交）。注意这里只是对不存在的数据添加默认值，没有验证数据类型等。 工作流程：向mysql的movie表中存入数据；向redis的tag_movie存入数据；向redis的actor_movie存入数据。 必须提供的数据有：no，即电影编号。未提供的部分不更新。 :param movie_dict: 包含movie数据的字典。""" conn, m_cur, r_cur, log = database_api_java.conn, self.m_cur, self.r_cur, self.log if len(movie_dict) <= 1: message = 'java_extensions.update_movie failed: movie_dict must include at least 2 items' log.warning(message) return try: no = movie_dict['no'] except: message = 'java_extensions.update_movie failed: no not offered' log.warning(message) return m_cur.execute('USE java;') command_movie = 'SELECT no FROM movie WHERE no={};'.format(no) m_cur.execute(command_movie) if len(m_cur.fetchall()) == 0: message = 'movie no.{} not exists'.format(no) log.warning(message) return command_movie = '''UPDATE movie SET ''' if 'code' in movie_dict.keys(): code = mysql.str_into_mysql(movie_dict['code']) command_movie += 'code="{}",'.format(code) if 'full_name' in movie_dict.keys(): full_name = mysql.str_into_mysql(movie_dict['full_name']) command_movie += 'full_name="{}",'.format(full_name) if 'release_date' in movie_dict.keys(): release_date = movie_dict['release_date'] command_movie += 'release_date="{}",'.format(release_date) if 'duration' in movie_dict.keys(): duration = movie_dict['duration'] command_movie += 'duration={},'.format(duration) if 'director' in movie_dict.keys(): director = mysql.str_into_mysql(movie_dict['director']) command_movie += 'director="{}",'.format(director) if 'producer' in movie_dict.keys(): producer = mysql.str_into_mysql(movie_dict['producer']) command_movie += 'producer="{}",'.format(producer) if 'publisher' in movie_dict.keys(): publisher = mysql.str_into_mysql(movie_dict['publisher']) command_movie += 'publisher="{}",'.format(publisher) if 'series' in movie_dict.keys(): series = mysql.str_into_mysql(movie_dict['series']) command_movie += 'series="{}",'.format(series) if 'url' in movie_dict.keys(): url = mysql.str_into_mysql(movie_dict['url']) command_movie += 'url="{}",'.format(url) if 'downloaded' in movie_dict.keys(): downloaded = movie_dict['downloaded'] command_movie += 'downloaded={},'.format(downloaded) if 'file_path' in movie_dict.keys(): file_path = mysql.str_into_mysql(movie_dict['file_path']) command_movie += 'file_path="{}",'.format(file_path) command_movie = command_movie[:-1] + ' WHERE no={};'.format(no) try: if command_movie.count('=') > 1: m_cur.execute(command_movie) conn.commit() pipe = r_cur.pipeline() if 'tags' in movie_dict.keys() and movie_dict['tags'] != None: pipe.srem('java:tag:no', str(no)) for tag in movie_dict['tags']: key = 'java:tag:{}'.format(tag) value = str(no) pipe.sadd(key, value) if 'actors' in movie_dict.keys() and movie_dict['actors'] != None: pipe.srem('java:actor:unknown', str(no)) for actor in movie_dict['actors']: key = 'java:actor:{}'.format(actor) value = str(no) pipe.sadd(key, value) pipe.execute() message = 'updated movie successfully, no={}'.format(no) log.info(message) except: conn.rollback() log.info('mysql错误命令为：' + command_movie) raise Exception('java_extensions.update_movie failed') def update_torrent(self, torrent_dict: dict): """更新一个torrent（以字典形式提交）。注意这里只是对不存在的数据添加默认值，没有验证数据类型等。 工作流程：向mysql的torrent表中存入数据；向redis的torrent_files存入数据。 必须提供的数据有：no和hash_value，即torrent编号及hash值。未提供的部分不更新。 :param torrent_dict: 包含torrent数据的字典。""" conn, m_cur, r_cur, log = database_api_java.conn, self.m_cur, self.r_cur, self.log if len(torrent_dict) <= 2: message = 'java_extensions.update_torrent failed: torrent_dict must include at least 3 items' log.warning(message) return try: no, hash_value = torrent_dict['no'], torrent_dict['hash_value'] except: message = 'java_extensions.update_torrent failed: no or hash_value not offered' log.warning(message) return m_cur.execute('USE java;') command_torrent = 'SELECT hash_value FROM torrent WHERE no={};'.format( no) m_cur.execute(command_torrent) if len(m_cur.fetchall()) == 0: message = 'torrent no.{} not exists'.format(no) log.warning(message) return if hash_value != m_cur.fetchall()[0][0]: message = 'torrent no.{} hash_value not match: {} versus {}'.format( no, hash_value, m_cur.fetchall()[0][0]) log.warning(message) return command_torrent = '''UPDATE torrent SET ''' if 'movie_no' in torrent_dict.keys(): movie_no = torrent_dict['movie_no'] command_torrent += 'movie_no={},'.format(movie_no) if 'name' in torrent_dict.keys(): name = mysql.str_into_mysql(torrent_dict['name']) command_torrent += 'name="{}",'.format(name) if 'file_num' in torrent_dict.keys(): file_num = torrent_dict['file_num'] command_torrent += 'file_num={},'.format(file_num) if 'size' in torrent_dict.keys(): size = torrent_dict['size'] command_torrent += 'size={},'.format(size) if 'high_definition' in torrent_dict.keys(): high_definition = torrent_dict['high_definition'] command_torrent += 'high_definition={},'.format(high_definition) if 'record_date' in torrent_dict.keys(): record_date = torrent_dict['record_date'] command_torrent += 'record_date="{}",'.format(record_date) command_torrent = command_torrent[:-1] + ' WHERE no={};'.format(no) try: if command_torrent.count('=') > 1: m_cur.execute(command_torrent) conn.commit() pipe = r_cur.pipeline() key = 'java:torrent:{}'.format(no) if 'torrent_files' in torrent_dict.keys(): for torrent_file in torrent_dict['torrent_files']: value = torrent_file pipe.sadd(key, value) pipe.execute() message = 'updated torrent successfully, no={}'.format(no) log.info(message) except: conn.rollback() log.info('mysql错误命令为：' + command_torrent) raise Exception('java_extensions.update_torrent failed') def check_repeat(self): """检查mysql的movie、torrent表里有没有重复的，只检查 (code和full_name) 、 (hash_value) 相同的。如果有重复的，保留no最小的一条记录。 检查redis的 java:actor: 、 java:tag: 、 java:torrent: ，是否对应的编号在mysql中还存在。""" conn, m_cur, r_cur, log = database_api_java.conn, self.m_cur, self.r_cur, self.log m_cur.execute('USE java;') #检查movie表 m_cur.execute( 'SELECT code,full_name FROM movie GROUP BY code,full_name HAVING COUNT(*)>1;' ) repeated_items = m_cur.fetchall() if len(repeated_items) == 0: message = 'no repeated code:full_name in movie' log.info(message) else: message = 'found repeated code:full_name {} kinds'.format( len(repeated_items)) log.info(message) command_movie = ''' DELETE FROM movie WHERE no IN (SELECT no FROM (SELECT no FROM movie WHERE (code,full_name) IN (SELECT code,full_name FROM movie GROUP BY code,full_name HAVING COUNT(*)>1) AND no NOT IN (SELECT MIN(no) FROM movie GROUP BY code,full_name HAVING COUNT(*)>1)) AS a);''' command_movie = command_movie.replace('\n', '') command_movie = command_movie.replace(' ', '') try: m_cur.execute(command_movie) conn.commit() message = 'repeated code:full_name deleted successfully. ' log.info(message) except: conn.rollback() log.info('mysql错误命令为：' + command_movie) raise Exception( 'java_extensions.check_repeat failed: table movie check_repeat failed' ) #检查torrent表 m_cur.execute( 'SELECT hash_value FROM torrent GROUP BY hash_value HAVING COUNT(*)>1;' ) repeated_items = m_cur.fetchall() if len(repeated_items) == 0: message = 'no repeated hash_value in torrent' log.info(message) else: message = 'found repeated hash_value {} kinds'.format( len(repeated_items)) log.info(message) command_torrent = ''' DELETE FROM torrent WHERE no IN (SELECT no FROM (SELECT no FROM torrent WHERE (hash_value) IN (SELECT hash_value FROM torrent GROUP BY hash_value HAVING COUNT(*)>1) AND no NOT IN (SELECT MIN(no) FROM torrent GROUP BY hash_value HAVING COUNT(*)>1)) AS a);''' command_torrent = command_torrent.replace('\n', '') command_torrent = command_torrent.replace(' ', '') try: m_cur.execute(command_torrent) conn.commit() command_torrent = 'DELETE FROM torrent WHERE hash_value="unknown";' m_cur.execute(command_torrent) conn.commit() message = 'repeated and unknown hash_value deleted successfully. ' log.info(message) except: conn.rollback() log.info('mysql错误命令为：' + command_torrent) raise Exception( 'java_extensions.check_repeat failed: table torrent check_repeat failed' ) try: pipe = r_cur.pipeline() #检查 java:actor:名字 actor_keys = r_cur.keys('java:actor:*') for actor_key in actor_keys: movie_nos = r_cur.smembers(actor_key) for movie_no in movie_nos: m_cur.execute( 'SELECT no FROM movie WHERE no={};'.format(movie_no)) if len(m_cur.fetchall()) == 0: pipe.srem(actor_key, movie_no) pipe.execute() #检查 java:tag:标签名 tag_keys = r_cur.keys('java:tag:*') for tag_key in tag_keys: movie_nos = r_cur.smembers(tag_key) for movie_no in movie_nos: m_cur.execute( 'SELECT no FROM movie WHERE no={};'.format(movie_no)) if len(m_cur.fetchall()) == 0: pipe.srem(tag_key, movie_no) pipe.execute() #检查 java:torrent:编号 torrent_keys = r_cur.keys('java:torrent:*') for torrent_key in torrent_keys: torrent_no = torrent_key.replace('java:torrent:', '') m_cur.execute( 'SELECT no FROM torrent WHERE no={};'.format(torrent_no)) if len(m_cur.fetchall()) == 0: pipe.delete(torrent_key) pipe.execute() except: raise Exception( 'java_extensions.check_repeat failed: redis check_repeat failed' ) def get_to_be_updated_torrents(self) -> list: '''对于未知name、file_num、文件名的torrent，取出它们。 :return [(no1,hash_value1), (no2,hash_value2), ...]''' m_cur, log = self.m_cur, self.log m_cur.execute('USE java;') command_torrent = '''SELECT no,hash_value FROM torrent WHERE name="unknown" UNION SELECT no,hash_value FROM torrent WHERE file_num=0 UNION SELECT no,hash_value FROM torrent WHERE size=0;''' command_torrent = command_torrent.replace('\n', '') command_torrent = command_torrent.replace(' ', '') try: m_cur.execute(command_torrent) message = 'to_be_updated_torrents selected. ' log.info(message) result = m_cur.fetchall() except: log.info('mysql错误命令为：' + command_torrent) raise Exception( 'java_extensions.get_to_be_updated_torrents failed: sql syntax wrong' ) return list(result) def torrent_selector(self, actors: list = None, tags: list = None, check_repeat=True, limit: int = 0): '''对电影、磁链进行筛选，每部电影筛选出一个最合适的磁链。结果保存在torrent_selector.txt中。 :param actors: 限定演员。多个演员时表示或(OR)，只要有一个演员出现即可。演员名为模糊搜索，只要包含即可 :param tags: 限定tag。多个tag时表示或(OR)，只要有一个标签出现即可 :param check_repeat: 是否去掉已经下载过的磁链。需要指定一个文件夹，该文件夹存放下载过的torrent文件。文件夹用TORRENT_DIR指定。 :param limit: 限定条数，0表示不限条数 演员和tag同时限定时，二者并不相关，就是说某部电影包含需要的演员、但不包含指定的tag（或者包含tag但不含演员），则该电影还是能被选上。''' conn, m_cur, r_cur, log = database_api_java.conn, self.m_cur, self.r_cur, self.log downloaded_hash = set() if check_repeat: for dirpath, dirnames, filenames in os.walk(TORRENT_DIR): for filename in filenames: abs_file_path = os.path.join(dirpath, filename) hash_value = torrent(abs_file_path).hash_value downloaded_hash.add(hash_value) print(f'\rdownloaded_hash added:{len(downloaded_hash)}', end='') m_cur.execute('USE java;') command_movie = 'SELECT no,release_date FROM movie WHERE downloaded=0 AND file_path="unknown"' movie_no_set = set() if actors != None: for actor in actors: actor_keys = r_cur.keys(f'java:actor:*{actor}*') for actor_key in actor_keys: movie_no = r_cur.smembers(actor_key) movie_no_set = movie_no_set.union(movie_no) if tags != None: for tag in tags: movie_no = r_cur.smembers(f'java:tag:{tag}') movie_no_set = movie_no_set.union(movie_no) if len(movie_no_set) != 0: movie_no_tuple = tuple(int(i) for i in movie_no_set) command_movie += ' AND no IN {}'.format(movie_no_tuple) if limit != 0: command_movie += f' LIMIT {limit}' command_movie += ';' m_cur.execute(command_movie) allowed_files = set() #allowed_files这个目录放的是一些文件，根据以往的经验，种子里带这些文件时电影的品质更好 for dirpath, dirnames, filenames in os.walk( os.path.join(os.getcwd(), 'allowed_files')): allowed_files.add(filenames) for movie_no, release_date in m_cur.fetchall(): command_torrent = 'SELECT * FROM torrent WHERE movie_no={};'.format( movie_no) m_cur.execute(command_torrent) torrent_tuples_list = list(m_cur.fetchall()) torrent_tuples_list.sort(key=lambda x: x[7]) #先根据record_date排序一次 torrent_score_list = [] for torrent_tuple in torrent_tuples_list: no, hash_value, movie_no, name, file_num, size, high_definition, record_date = torrent_tuple torrent_files = r_cur.smembers('java:torrent:{}'.format(no)) score = 0 #根据name对torrent进行打分，这也是最主要的打分手段 p_thz = re.compile('thz', flags=re.I) p_hnd = re.compile('hnd', flags=re.I) p_C = re.compile('CH?$', flags=re.I) p_HD = re.compile('(HD$)|(^HD)', flags=re.I) p_2048 = re.compile('^hjd-2048', flags=re.I) p_zip = re.compile('\.zip$', flags=re.I) p_mp4 = re.compile('[^C ]\.mp4$', flags=re.I) if re.search(p_thz, name) != None or re.search( p_hnd, name) != None or ("原版首发" in name) or re.search( p_mp4, name) != None: score += 50 if re.search(p_C, name) != None or re.search( p_2048, name) != None or re.search( p_zip, name) != None or re.search( p_HD, name) != None or name.startswith( '-') or name.startswith( '@') or name.startswith('_'): score -= 50 #对中文字幕的要求不是很强烈（而且中文字幕的经常带广告），所以这里是扣分项 letters = re.findall('[a-zA-Z]+', name) all_upper_letter = True for letter in letters: if letter == 'mp' or letter.isupper(): continue else: all_upper_letter = False break if all_upper_letter == True: score += 30 #file_num if file_num == 1: score += 100 #high_definition if high_definition == 1: score += 50 #record_date #如果电影早于2018年，这时的磁链以能下载为优先，时间近的分数更高；晚于2018年的电影则时间越早分数越高，因为晚出来的磁链经常被人处理过 #这里没有分的太细，比如有些磁链的日期相同，却没有给相同的分数 if release_date > datetime.date(2018, 1, 1): score += 100 / len(torrent_tuples_list) * ( len(torrent_tuples_list) - 1 - torrent_tuples_list.index(torrent_tuple)) else: score += 100 / len(torrent_tuples_list) * ( 1 + torrent_tuples_list.index(torrent_tuple)) #torrent_files for torrent_file in torrent_files: if torrent_file in allowed_files or 'hnd' in torrent_file: score += 100 break torrent_score_list.append((score, name, hash_value)) torrent_score_list.sort(key=lambda x: x[0]) if len(torrent_score_list) != 0: score, name, hash_value = torrent_score_list[-1] hash_value = hash_value.upper() if check_repeat: if hash_value not in downloaded_hash: with open(os.path.join(os.getcwd(), 'torrent_selector.txt'), 'a', encoding='utf-8') as f: f.write( f'magnet:?xt=urn:btih:{hash_value}&dn={name}' + '\n') else: message = f'torrent file downloaded, hash_value={hash_value}' log.info(message) else: with open(os.path.join(os.getcwd(), 'torrent_selector.txt'), 'a', encoding='utf-8') as f: f.write(f'magnet:?xt=urn:btih:{hash_value}&dn={name}' + '\n') else: message = f'cannot find torrent for movie_no={movie_no}' log.warning(message) if __name__ == '__main__': actors = ['1', '2', '3'] tags = [] database_api_java().torrent_selector(actors=actors, tags=tags, limit=0) 本来还有一个 btsow_extensions.py ，它的作用是查询一些磁链搜索网站，这些网站能根据hash值查询磁链的信息，但是这些网站的反爬比较严格，这个文件和对应的爬虫这里就不写了。因此，实际上 get_to_be_updated_torrents 这个方法没有用到，根据磁链内文件名对磁链打分的功能也是缺失的。3.scrapy项目的修改（1）items.py# Define here the models for your scraped items # # See documentation in: # https://docs.scrapy.org/en/latest/topics/items.html import scrapy class JavaspiderItem(scrapy.Item): # define the fields for your item here like: # name = scrapy.Field() movie_dict = scrapy.Field() torrent_dicts_list = scrapy.Field() class TorrentDetailItem(scrapy.Item): torrent_dict = scrapy.Field()（2）middlewares.py# Define here the models for your spider middleware # # See documentation in: # https://docs.scrapy.org/en/latest/topics/spider-middleware.html from scrapy import signals # useful for handling different item types with a single interface from itemadapter import is_item, ItemAdapter class JavaspiderSpiderMiddleware: # Not all methods need to be defined. If a method is not defined, # scrapy acts as if the spider middleware does not modify the # passed objects. @classmethod def from_crawler(cls, crawler): # This method is used by Scrapy to create your spiders. s = cls() crawler.signals.connect(s.spider_opened, signal=signals.spider_opened) return s def process_spider_input(self, response, spider): # Called for each response that goes through the spider # middleware and into the spider. # Should return None or raise an exception. return None def process_spider_output(self, response, result, spider): # Called with the results returned from the Spider, after # it has processed the response. # Must return an iterable of Request, or item objects. for i in result: yield i def process_spider_exception(self, response, exception, spider): # Called when a spider or process_spider_input() method # (from other spider middleware) raises an exception. # Should return either None or an iterable of Request or item objects. pass def process_start_requests(self, start_requests, spider): # Called with the start requests of the spider, and works # similarly to the process_spider_output() method, except # that it doesn’t have a response associated. # Must return only requests (not items). for r in start_requests: yield r def spider_opened(self, spider): spider.logger.info('Spider opened: %s' % spider.name) import random class JavaspiderDownloaderMiddleware: # Not all methods need to be defined. If a method is not defined, # scrapy acts as if the downloader middleware does not modify the # passed objects. UA_list = [ 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36', 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36 Edg/105.0.1343.33', 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:104.0) Gecko/20100101 Firefox/104.0' ] @classmethod def from_crawler(cls, crawler): # This method is used by Scrapy to create your spiders. s = cls() crawler.signals.connect(s.spider_opened, signal=signals.spider_opened) return s def process_request(self, request, spider): # Called for each request that goes through the downloader # middleware. # Must either: # - return None: continue processing this request # - or return a Response object # - or return a Request object # - or raise IgnoreRequest: process_exception() methods of # installed downloader middleware will be called request.headers['User-Agent'] = random.choice(self.UA_list) return None def process_response(self, request, response, spider): # Called with the response returned from the downloader. # Must either; # - return a Response object # - return a Request object # - or raise IgnoreRequest return response def process_exception(self, request, exception, spider): # Called when a download handler or a process_request() # (from other downloader middleware) raises an exception. # Must either: # - return None: continue processing this exception # - return a Response object: stops process_exception() chain # - return a Request object: stops process_exception() chain pass def spider_opened(self, spider): spider.logger.info('Spider opened: %s' % spider.name) （3）pipelines.py# Define your item pipelines here # # Don't forget to add your pipeline to the ITEM_PIPELINES setting # See: https://docs.scrapy.org/en/latest/topics/item-pipeline.html # useful for handling different item types with a single interface from itemadapter import ItemAdapter from javaspider.libs.java_extensions import database_api_java class JavaspiderPipeline: def open_spider(self, spider): self.database_api_java = database_api_java() def process_item(self, item, spider): movie_no = '0' if 'movie_dict' in item.keys(): movie_no = self.database_api_java.save_movie(item['movie_dict']) if 'torrent_dicts_list' in item.keys(): for torrent_dict in item['torrent_dicts_list']: if movie_no != None: torrent_dict['movie_no'] = int(movie_no) self.database_api_java.save_torrent(torrent_dict) if 'torrent_dict' in item.keys(): #self.database_api_java.update_torrent(torrent_dict) pass return item def close_spider(self, spider): self.database_api_java.check_repeat()（4）settings.py# Scrapy settings for javaspider project # # For simplicity, this file contains only settings considered important or # commonly used. You can find more settings consulting the documentation: # # https://docs.scrapy.org/en/latest/topics/settings.html # https://docs.scrapy.org/en/latest/topics/downloader-middleware.html # https://docs.scrapy.org/en/latest/topics/spider-middleware.html LOG_LEVEL = 'ERROR' BOT_NAME = 'javaspider' SPIDER_MODULES = ['javaspider.spiders'] NEWSPIDER_MODULE = 'javaspider.spiders' # Crawl responsibly by identifying yourself (and your website) on the user-agent #USER_AGENT = 'javaspider (+http://www.yourdomain.com)' # Obey robots.txt rules ROBOTSTXT_OBEY = False # Configure maximum concurrent requests performed by Scrapy (default: 16) #CONCURRENT_REQUESTS = 32 # Configure a delay for requests for the same website (default: 0) # See https://docs.scrapy.org/en/latest/topics/settings.html#download-delay # See also autothrottle settings and docs #DOWNLOAD_DELAY = 3 # The download delay setting will honor only one of: #CONCURRENT_REQUESTS_PER_DOMAIN = 16 #CONCURRENT_REQUESTS_PER_IP = 16 # Disable cookies (enabled by default) #COOKIES_ENABLED = False # Disable Telnet Console (enabled by default) #TELNETCONSOLE_ENABLED = False # Override the default request headers: #DEFAULT_REQUEST_HEADERS = { # 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', # 'Accept-Language': 'en', #} # Enable or disable spider middlewares # See https://docs.scrapy.org/en/latest/topics/spider-middleware.html #SPIDER_MIDDLEWARES = { # 'javaspider.middlewares.JavaspiderSpiderMiddleware': 543, #} # Enable or disable downloader middlewares # See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html DOWNLOADER_MIDDLEWARES = { 'javaspider.middlewares.JavaspiderDownloaderMiddleware': 543, } # Enable or disable extensions # See https://docs.scrapy.org/en/latest/topics/extensions.html #EXTENSIONS = { # 'scrapy.extensions.telnet.TelnetConsole': None, #} # Configure item pipelines # See https://docs.scrapy.org/en/latest/topics/item-pipeline.html ITEM_PIPELINES = { 'javaspider.pipelines.JavaspiderPipeline': 300, } # Enable and configure the AutoThrottle extension (disabled by default) # See https://docs.scrapy.org/en/latest/topics/autothrottle.html #AUTOTHROTTLE_ENABLED = True # The initial download delay #AUTOTHROTTLE_START_DELAY = 5 # The maximum download delay to be set in case of high latencies #AUTOTHROTTLE_MAX_DELAY = 60 # The average number of requests Scrapy should be sending in parallel to # each remote server #AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0 # Enable showing throttling stats for every response received: #AUTOTHROTTLE_DEBUG = False # Enable and configure HTTP caching (disabled by default) # See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html#httpcache-middleware-settings #HTTPCACHE_ENABLED = True #HTTPCACHE_EXPIRATION_SECS = 0 #HTTPCACHE_DIR = 'httpcache' #HTTPCACHE_IGNORE_HTTP_CODES = [] #HTTPCACHE_STORAGE = 'scrapy.extensions.httpcache.FilesystemCacheStorage' （5）爬虫爬虫文件名为 java.py ，在之前已经创建过了，内容为：import scrapy, re, random, os, time import urllib.parse from javaspider.items import JavaspiderItem from javaspider.libs import java_extensions as j_e from MyPythonLib import interaction error_html_file_path = os.path.join(os.getcwd(), 'wrong_response_java.html') #这个html文件用于存储解析出错的页面，便于查找问题 RECURSIVE=True #如果演员主页都是第一次爬，那么就打开递归爬取的功能，自动爬取下一页；对于以前曾经爬过的演员，并且间隔时间不久（3个月之内），则只爬取第一页就行了，把这个值设为False。 class JavaSpider(scrapy.Spider): name = 'java' start_urls = j_e.get_actor_main_pages() custom_settings = custom_settings = { 'CONCURRENT_REQUESTS': 16, 'DOWNLOAD_DELAY': 2, 'CONCURRENT_REQUESTS_PER_DOMAIN': 1 }#控制爬取速度 origin_url = j_e.ORIGIN_URL_JAVA temp_url = j_e.TEMP_URL_JAVA start_time = time.time() total = len(start_urls) progress = 0 msg = '' def parse(self, response): try: movie_nodes = response.xpath( '//*[@id="waterfall"]/div/a[@class="movie-box" and @href]') for movie_node in movie_nodes: detail_url = movie_node.xpath('./@href').get() if isinstance(detail_url, str): detail_url = urllib.parse.urljoin(JavaSpider.temp_url, detail_url.strip()) movie_dict = {} code = movie_node.xpath('.//date[1]/text()').get() if isinstance(code, str): movie_dict['code'] = code.strip() full_name = movie_node.xpath('.//span/text()[1]').get() if isinstance(full_name, str): movie_dict['full_name'] = full_name.strip() release_date = movie_node.xpath('.//date[2]/text()').get() if isinstance(release_date, str): movie_dict['release_date'] = release_date.strip() movie_dict['url'] = detail_url.replace(JavaSpider.temp_url, JavaSpider.origin_url) item = JavaspiderItem() item['movie_dict'] = movie_dict request = scrapy.Request(url=detail_url, callback=self.parse_movie) request.cb_kwargs['item'] = item yield request next_url = response.xpath( '//*[@id="next" and text()="下一頁"]/@href').get() if RECURSIVE and isinstance(next_url, str): next_url = urllib.parse.urljoin(JavaSpider.temp_url, next_url.strip()) yield scrapy.Request(url=next_url, callback=self.parse) if response.url in JavaSpider.start_urls: JavaSpider.progress = 1 + JavaSpider.start_urls.index( response.url) JavaSpider.msg = response.url.replace( JavaSpider.temp_url, '') except: with open(error_html_file_path, 'a', encoding='utf-8') as f: f.write(response.text) message = 'JavaSpider.parse failed' raise Exception(message) def parse_movie(self, response, item): try: movie_dict = item['movie_dict'] info_node = response.xpath('/html/body/div[5]/div[1]/div[2]') duration = info_node.xpath( './p/span[text()="長度:"]/../text()').get() if isinstance(duration, str): duration = re.search('\d+', duration.strip()).group() movie_dict['duration'] = duration director = info_node.xpath( './p/span[text()="導演:"]/../a/text()').get() if isinstance(director, str): movie_dict['director'] = director.strip() producer = info_node.xpath( './p/span[text()="製作商:"]/../a/text()').get() if isinstance(producer, str): movie_dict['producer'] = producer.strip() publisher = info_node.xpath( './p/span[text()="發行商:"]/../a/text()').get() if isinstance(publisher, str): movie_dict['publisher'] = publisher.strip() series = info_node.xpath( './p/span[text()="系列:"]/../a/text()').get() if isinstance(series, str): movie_dict['series'] = series.strip() movie_dict['tags'] = info_node.xpath( './p[text()="類別:"]/following-sibling::p[1]/span//a/text()' ).getall() if movie_dict['tags'] == None or len(movie_dict['tags']) == 0: movie_dict['tags'] = set(['no']) movie_dict['actors'] = info_node.xpath( './p/span[text()="演員"]/../following-sibling::p[1]/span/a/text()' ).getall() if movie_dict['actors'] == None or len(movie_dict['actors']) == 0: movie_dict['actors'] = set(['unknown']) js = response.xpath(r'/html/body/script[3]/text()').get().strip() gid = re.search(r'\d+', re.search(r'gid.+;', js).group()).group() uc = re.search(r'\d+', re.search(r'uc.+;', js).group()).group() img = re.search(r"""['|"].+['|"]""", re.search(r'img.+;', js).group()).group()[1:-1] torrents_url = urllib.parse.urljoin(JavaSpider.temp_url, 'ajax/uncledatoolsbyajax.php') torrents_url = torrents_url + '?gid=' + gid + '&lang=zh&img=' + img + '&uc=' + uc + '&floor=' + str( random.randint(10, 999)) request = scrapy.Request(url=torrents_url, callback=self.parse_torrents) interaction.print_progress(self.start_time, self.progress, self.total, self.msg) request.cb_kwargs['item'] = item yield request except: with open(error_html_file_path, 'a', encoding='utf-8') as f: f.write(response.text) message = 'JavaSpider.parse_movie failed' raise Exception(message) def parse_torrents(self, response, item): try: torrent_dicts_list = [] torrent_nodes = response.xpath('//tr') for torrent_node in torrent_nodes: torrent_dict = {} hash_value = torrent_node.xpath('./td[1]/a/@href').get() if isinstance(hash_value, str): hash_value = hash_value.strip().replace( r'magnet:?xt=urn:btih:', '') torrent_dict['hash_value'] = hash_value.split('&')[0] name = torrent_node.xpath('./td[1]/a/text()').get() if isinstance(name, str): torrent_dict['name'] = name.strip() size = torrent_node.xpath('./td[2]/a/text()').get() #该网站的磁链size时有错误，常见的有：无单位，有2个小数点等 if isinstance(size, str): size = size.strip() p_gb = re.compile(r'gb$', flags=re.I) p_mb = re.compile(r'mb$', flags=re.I) p_tb = re.compile(r'tb$', flags=re.I) p_kb = re.compile(r'kb$', flags=re.I) if re.search(p_gb, size) != None: size = int(1024 * float(re.search(r'[\d|\.]+', size).group())) elif re.search(p_mb, size) != None: size = int(float(re.search(r'[\d|\.]+', size).group())) elif re.search(p_tb, size) != None: size = int(1024 * 1024 * float(re.search(r'[\d|\.]+', size).group())) elif re.search(p_kb, size) != None: size = int( float(re.search(r'[\d|\.]+', size).group()) / 1024) elif size == '': message = 'JavaSpider.parse_torrents warning: get size from empty string "{}"'.format( size) print(message) size = '0' else: message = 'JavaSpider.parse_torrents warning: size no unit "{}"'.format( size) print(message) try: size = int( 1024 * float(re.search(r'[\d|\.]+', size).group())) except: message = 'JavaSpider.parse_torrents warning: size wrong "{}"'.format( size) print(message) size = '0' torrent_dict['size'] = size record_date = torrent_node.xpath('./td[3]/a/text()').get() if isinstance(record_date, str): torrent_dict['record_date'] = record_date.strip() high_definition = torrent_node.xpath('.//a[text()="高清"]') torrent_dict['high_definition'] = ( 0, 1)[len(high_definition) != 0] torrent_dicts_list.append(torrent_dict) item['torrent_dicts_list'] = torrent_dicts_list yield item except: with open(error_html_file_path, 'a', encoding='utf-8') as f: f.write(response.text) message = 'JavaSpider.parse_torrents failed' raise Exception(message)4.使用说明首先用chrome或其他浏览器访问该电影网站，挑选自己偏好的演员，演员的主页保存到一个书签，比如说可以每50个演员主页放到一个书签文件夹中。要爬取时，进入书签管理器，用 ctrl+A 选中并复制一个书签文件夹内的所有演员主页，把这些网址复制到 D:\spiders\scrapy\javaspider\javaspider\actor_main_pages.txt 里，每行一个url。然后考虑一下，如果这些演员在3个月之内爬过，就到爬虫文件 java.py 里把 RECURSIVE = True 改成 False ，然后检查一下mysql和redis是否打开，最后在cmd中如下运行爬虫：D: cd D:\spiders\scrapy\javaspider\javaspider scrapy crawl java应该能看到一个图形化的进度显示条。等待爬取完成，按照笔者的经验，使用前文中限制爬虫的配置： custom_settings = custom_settings = { 'CONCURRENT_REQUESTS': 16, 'DOWNLOAD_DELAY': 2, 'CONCURRENT_REQUESTS_PER_DOMAIN': 1 }这样的话，爬取50个演员主页，大概需要4小时。爬取完成后，登录mysql和redis检查一下数据。对于redis-cli，记得添加 --raw 参数来显示utf-8字符，比如中文、日文字符。最后，在 java_extensions.py 的最后，有调用 torrent_selector 方法，调整一下演员、tag并运行该文件，会生成 D:\spiders\scrapy\javaspider\torrent_selector.txt 文件，里面是为每一部电影筛选出的磁链，每个磁链占一行，可以非常方便地复制到qbittorrent或aria2等客户端内进行下载（为了便捷性，不推荐使用迅雷下载）。至此，本文中完成了电影信息爬取（根据演员主页），磁链爬取，磁链筛选与下载。下一篇文章中将介绍对本地电影进行入库，和对电影进行一定压缩，以节约空间。
本文较长，属于系统性总结，可作为使用手册查阅。步骤/目录： 1.Debugging Spiders 2.Common Practices 3.Broad Crawls 4.使用浏览器的开发者工具 5.下载图片与文件 6.AutoThrottle extension 7.Benchmarking 8.爬虫的暂停与恢复 9.scrapy架构 10.下载中间件 11.爬虫中间件 12.添加extensions 13.其他本文首发于个人博客https://lisper517.top/index.php/archives/52/，转载请注明出处。本文的目的是详细介绍scrapy，主要翻译自 scrapy官方文档 。本文写作日期为2022年9月15日。运行的平台为win10，编辑器为VS Code。在上一篇文章中，介绍了scrapy的基础概念，现在讲解一些进阶的用法和技巧。1.Debugging Spiders介绍了一些调试爬虫的方式，主要有：（1） scrapy parse 命令。（2） scrapy.shell 类，使用例：from scrapy.shell import inspect_response def parse_details(self, response, item=None): if item: # populate more `item` fields return item else: inspect_response(response, self)（3） scrapy.utils.response 类，可以打开浏览器：from scrapy.utils.response import open_in_browser def parse_details(self, response): if "item name" not in response.body: open_in_browser(response)（4）Logging。（5）Spiders Contracts。2.Common Practices如何使用爬虫。内容包括：（1）使用 scrapy crawl 以外的方式运行爬虫，比如运行多个爬虫或全部爬虫：from scrapy.crawler import CrawlerProcess from scrapy.utils.project import get_project_settings from scrapy.spiderloader import SpiderLoader # 根据项目配置获取 CrawlerProcess 实例 process = CrawlerProcess(get_project_settings()) # 获取 spiderloader 对象，以进一步获取项目下所有爬虫名称 spider_loader = SpiderLoader(get_project_settings()) # 添加全部爬虫 for spidername in spider_loader.list(): process.crawl(spidername) # 添加一个爬虫 #process.crawl('爬虫名') # 执行 process.start()（2）分布式爬取，涉及scrapyd。（3）避免被ban ip，原文的建议有：rotate your user agent from a pool of well-known ones from browsers (google around to get a list of them) disable cookies (see COOKIES_ENABLED) as some sites may use cookies to spot bot behaviour use download delays (2 or higher). See DOWNLOAD_DELAY setting. if possible, use Google cache to fetch pages, instead of hitting the sites directly use a pool of rotating IPs. For example, the free Tor project or paid services like ProxyMesh. An open source alternative is scrapoxy, a super proxy that you can attach your own proxies to. use a highly distributed downloader that circumvents bans internally, so you can just focus on parsing clean pages. One example of such downloaders is Zyte Smart Proxy Manager3.Broad Crawls目前为止介绍的都是focus爬虫，即聚焦于某一个站点。如果是google、baidu等搜索引擎，它们会用Broad爬虫，这部分介绍了如何把scrapy改造成Broad爬虫，有兴趣的可阅读原文。Scrapy defaults are optimized for crawling specific sites. These sites are often handled by a single Scrapy spider, although this is not necessary or required (for example, there are generic spiders that handle any given site thrown at them). In addition to this “focused crawl”, there is another common type of crawling which covers a large (potentially unlimited) number of domains, and is only limited by time or other arbitrary constraint, rather than stopping when the domain was crawled to completion or when there are no more requests to perform. These are called “broad crawls” and is the typical crawlers employed by search engines. These are some common properties often found in broad crawls: they crawl many domains (often, unbounded) instead of a specific set of sites they don’t necessarily crawl domains to completion, because it would be impractical (or impossible) to do so, and instead limit the crawl by time or number of pages crawled they are simpler in logic (as opposed to very complex spiders with many extraction rules) because data is often post-processed in a separate stage they crawl many domains concurrently, which allows them to achieve faster crawl speeds by not being limited by any particular site constraint (each site is crawled slowly to respect politeness, but many sites are crawled in parallel) As said above, Scrapy default settings are optimized for focused crawls, not broad crawls. However, due to its asynchronous architecture, Scrapy is very well suited for performing fast broad crawls. This page summarizes some things you need to keep in mind when using Scrapy for doing broad crawls, along with concrete suggestions of Scrapy settings to tune in order to achieve an efficient broad crawl. Use the right SCHEDULER_PRIORITY_QUEUE¶ Scrapy’s default scheduler priority queue is 'scrapy.pqueues.ScrapyPriorityQueue'. It works best during single-domain crawl. It does not work well with crawling many different domains in parallel To apply the recommended priority queue use: SCHEDULER_PRIORITY_QUEUE = 'scrapy.pqueues.DownloaderAwarePriorityQueue' Increase concurrency¶ Concurrency is the number of requests that are processed in parallel. There is a global limit (CONCURRENT_REQUESTS) and an additional limit that can be set either per domain (CONCURRENT_REQUESTS_PER_DOMAIN) or per IP (CONCURRENT_REQUESTS_PER_IP). Note The scheduler priority queue recommended for broad crawls does not support CONCURRENT_REQUESTS_PER_IP. The default global concurrency limit in Scrapy is not suitable for crawling many different domains in parallel, so you will want to increase it. How much to increase it will depend on how much CPU and memory you crawler will have available. A good starting point is 100: CONCURRENT_REQUESTS = 100 But the best way to find out is by doing some trials and identifying at what concurrency your Scrapy process gets CPU bounded. For optimum performance, you should pick a concurrency where CPU usage is at 80-90%. Increasing concurrency also increases memory usage. If memory usage is a concern, you might need to lower your global concurrency limit accordingly. Increase Twisted IO thread pool maximum size¶ Currently Scrapy does DNS resolution in a blocking way with usage of thread pool. With higher concurrency levels the crawling could be slow or even fail hitting DNS resolver timeouts. Possible solution to increase the number of threads handling DNS queries. The DNS queue will be processed faster speeding up establishing of connection and crawling overall. To increase maximum thread pool size use: REACTOR_THREADPOOL_MAXSIZE = 20 Setup your own DNS¶ If you have multiple crawling processes and single central DNS, it can act like DoS attack on the DNS server resulting to slow down of entire network or even blocking your machines. To avoid this setup your own DNS server with local cache and upstream to some large DNS like OpenDNS or Verizon. Reduce log level¶ When doing broad crawls you are often only interested in the crawl rates you get and any errors found. These stats are reported by Scrapy when using the INFO log level. In order to save CPU (and log storage requirements) you should not use DEBUG log level when preforming large broad crawls in production. Using DEBUG level when developing your (broad) crawler may be fine though. To set the log level use: LOG_LEVEL = 'INFO' Disable cookies¶ Disable cookies unless you really need. Cookies are often not needed when doing broad crawls (search engine crawlers ignore them), and they improve performance by saving some CPU cycles and reducing the memory footprint of your Scrapy crawler. To disable cookies use: COOKIES_ENABLED = False Disable retries¶ Retrying failed HTTP requests can slow down the crawls substantially, specially when sites causes are very slow (or fail) to respond, thus causing a timeout error which gets retried many times, unnecessarily, preventing crawler capacity to be reused for other domains. To disable retries use: RETRY_ENABLED = False Reduce download timeout¶ Unless you are crawling from a very slow connection (which shouldn’t be the case for broad crawls) reduce the download timeout so that stuck requests are discarded quickly and free up capacity to process the next ones. To reduce the download timeout use: DOWNLOAD_TIMEOUT = 15 Disable redirects¶ Consider disabling redirects, unless you are interested in following them. When doing broad crawls it’s common to save redirects and resolve them when revisiting the site at a later crawl. This also help to keep the number of request constant per crawl batch, otherwise redirect loops may cause the crawler to dedicate too many resources on any specific domain. To disable redirects use: REDIRECT_ENABLED = False Enable crawling of “Ajax Crawlable Pages”¶ Some pages (up to 1%, based on empirical data from year 2013) declare themselves as ajax crawlable. This means they provide plain HTML version of content that is usually available only via AJAX. Pages can indicate it in two ways: by using #! in URL - this is the default way; by using a special meta tag - this way is used on “main”, “index” website pages. Scrapy handles (1) automatically; to handle (2) enable AjaxCrawlMiddleware: AJAXCRAWL_ENABLED = True When doing broad crawls it’s common to crawl a lot of “index” web pages; AjaxCrawlMiddleware helps to crawl them correctly. It is turned OFF by default because it has some performance overhead, and enabling it for focused crawls doesn’t make much sense. Crawl in BFO order¶ Scrapy crawls in DFO order by default. In broad crawls, however, page crawling tends to be faster than page processing. As a result, unprocessed early requests stay in memory until the final depth is reached, which can significantly increase memory usage. Crawl in BFO order instead to save memory. Be mindful of memory leaks¶ If your broad crawl shows a high memory usage, in addition to crawling in BFO order and lowering concurrency you should debug your memory leaks. Install a specific Twisted reactor¶ If the crawl is exceeding the system’s capabilities, you might want to try installing a specific Twisted reactor, via the TWISTED_REACTOR setting.4.使用浏览器的开发者工具永远不要在xpath表达式里带tbody标签。浏览器对页面的源码会做一些修改（排版美化，执行js）、scrapy不会，这将导致浏览器看到的源码和scrapy中不同，其中一个点就是firefox浏览器喜欢在表格中加tbody标签。其他的部分笔者也不介绍了，其他地方有更详细的使用开发者工具教程，最好是找一些图文教程或视频教程看。5.下载图片与文件使用image管道和files管道。它们的特点有：避免重复下载（会scrapy会给文件生成hash身份码用于去重），指定下载地点（本地目录，FTP服务器，google云）。image管道还有另外的特点，包括自动转换图片为jpg格式、RGB模式（这对下载gif图片有些麻烦），生成图片指纹（thumbnail），对下载图片的长、宽进行限制。官方文档关于使用Files管道的流程解说：The typical workflow, when using the FilesPipeline goes like this: In a Spider, you scrape an item and put the URLs of the desired into a file_urls field. The item is returned from the spider and goes to the item pipeline. When the item reaches the FilesPipeline, the URLs in the file_urls field are scheduled for download using the standard Scrapy scheduler and downloader (which means the scheduler and downloader middlewares are reused), but with a higher priority, processing them before other pages are scraped. The item remains “locked” at that particular pipeline stage until the files have finish downloading (or fail for some reason). When the files are downloaded, another field (files) will be populated with the results. This field will contain a list of dicts with information about the downloaded files, such as the downloaded path, the original scraped url (taken from the file_urls field), the file checksum and the file status. The files in the list of the files field will retain the same order of the original file_urls field. If some file failed downloading, an error will be logged and the file won’t be present in the files field.关于Images管道的介绍：Using the ImagesPipeline is a lot like using the FilesPipeline, except the default field names used are different: you use image_urls for the image URLs of an item and it will populate an images field for the information about the downloaded images. The advantage of using the ImagesPipeline for image files is that you can configure some extra functions like generating thumbnails and filtering the images based on their size. The Images Pipeline requires Pillow 4.0.0 or greater. It is used for thumbnailing and normalizing images to JPEG/RGB format.使用Media管道的流程如下：（1）在settings.py中修改默认的ITEM_PIPELINES，加上Media管道。然后设置存储目录（不设置存储目录的话也不会下载），比如：FILES_STORE = '/path/to/valid/dir' IMAGES_STORE = '/path/to/valid/dir'（2）指定文件名。默认情况下，文件名是以其来源网址进行 SHA-1 hash 后得到的字符串为名，如果是image则加上jpg后缀。但是你也可以在Media管道类中重写 file_path() 方法来自己命名文件，比如对 http://www.example.com/product/images/large/front/0000000004166 ，想命名为 00b08510e4_front.jpg （需要hash一下）：import hashlib from os.path import splitext def file_path(self, request, response=None, info=None, *, item=None): image_url_hash = hashlib.shake_256(request.url.encode()).hexdigest(5) image_perspective = request.url.split('/')[-2] image_filename = f'{image_url_hash}_{image_perspective}.jpg' return image_filename自己指定文件名最重要的原则是避免重名，因为这时旧的文件会被覆盖。同时，重写 file_path() 方法也可以用来将图片以原来的格式保存，这一点在保存gif文件时尤其有用，具体见后。（3）存储。支持的存储有本地存储，FTP服务器，Amazon S3存储，google云，这里只介绍本地存储，以image来说，它会被存储到：<IMAGES_STORE>/full/<FILE_NAME>这个目录下。其中full是因为要把thumbnail和图片分开存储而存在的。关于FTP服务器存储，请看原文：New in version 2.0. FILES_STORE and IMAGES_STORE can point to an FTP server. Scrapy will automatically upload the files to the server. FILES_STORE and IMAGES_STORE should be written in one of the following forms: ftp://username:password@address:port/path ftp://address:port/path If username and password are not provided, they are taken from the FTP_USER and FTP_PASSWORD settings respectively. FTP supports two different connection modes: active or passive. Scrapy uses the passive connection mode by default. To use the active connection mode instead, set the FEED_STORAGE_FTP_ACTIVE setting to True.修改文件的流程如下：（1）在settings.py里打开Media管道，并用 FILES_STORE 和 IMAGES_STORE 设置存放目录。（2）在items.py里添加专门爬取Media的field，比如：import scrapy class MyItem(scrapy.Item): # ... other item fields ... image_urls = scrapy.Field() images = scrapy.Field()image_urls 和 images 也可以在settings中更改：IMAGES_URLS_FIELD = 'field_name_for_your_images_urls' IMAGES_RESULT_FIELD = 'field_name_for_your_processed_images' FILES_URLS_FIELD = 'field_name_for_your_files_urls' FILES_RESULT_FIELD = 'field_name_for_your_processed_files'后面两个是Files管道的。如果你的Media管道不是默认名，则在前面加上该名称，比如你的Images管道类叫做 MyPipeline ，那么在settings里需要指定为：MYPIPELINE_IMAGES_URLS_FIELD = MYPIPELINE_IMAGES_RESULT_FIELD = MYPIPELINE_FILES_URLS_FIELD = MYPIPELINE_FILES_RESULT_FIELD = 最后，讲解一些其余特征：（1）对近期（90天）内下载过的文件或图片，scrapy不会重复下载，这段时间的长短可以在settings.py中调整：# 120 days of delay for files expiration FILES_EXPIRES = 120 MYPIPELINE_FILES_EXPIRES = 120 # 30 days of delay for images expiration IMAGES_EXPIRES = 30 MYPIPELINE_IMAGES_EXPIRES = 30注意它们的单位是天。scrapy会将文件最后修改时间与现在时间比较，来判断一个文件是否在这个下载保护期内。（2）设置图片指纹，如：IMAGES_THUMBS = { 'small': (50, 50), 'big': (270, 270), }这里的键是指纹名，后面则是指纹图片长宽。如上设置时，一张图片会存储三个文件：<IMAGES_STORE>/full/图片名.jpg <IMAGES_STORE>/thumbs/small/图片名.jpg <IMAGES_STORE>/thumbs/big/图片名.jpg（3）过滤小图片。格式如下：IMAGES_MIN_HEIGHT = 110 IMAGES_MIN_WIDTH = 110如果对长宽都有限制，则图片要满足这两个限制才会被下载。默认情况下没有图片长宽限制。（4）允许重定向。默认情况下，Media管道发出的请求被重定向时scrapy会认为下载失败了。使用：MEDIA_ALLOW_REDIRECTS = True则允许Media管道下载重定向。重写Media管道类的方法，对于Files管道类，方法有：（1） file_path(self, request, response=None, info=None, *, item=None) ，这个方法每次下载item时会调用一次，用于给出存储的文件名。默认情况下它返回的是 full/<request URL hash>.<extension> ，前面要加上settings.py里设置的目录才是绝对路径。比如对于 https://example.com/a/b/c/foo.png 这张图片，使用例如下：import os from urllib.parse import urlparse from scrapy.pipelines.files import FilesPipeline class MyFilesPipeline(FilesPipeline): def file_path(self, request, response=None, info=None, *, item=None): return 'files/' + os.path.basename(urlparse(request.url).path)（2） get_media_requests(item, info) ，前面提到过，管道类会通过item拿到需要下载的url，然后会使用这个方法生成request对象去下载文件。这些request对象完成下载后会以2元素元组的列表格式发送给下面提到的 item_completed() 方法，每个2元素元组的格式为 (success, file_info_or_error) ，前者是bool，表示是否下载成功，后者是一个字典（如果下载成功），包含下载的一些信息，或者是一个twisted的Failure对象（如果下载失败）。另外注意， item_completed() 方法收到的元组列表的顺序和 get_media_requests(item, info) 发出的请求顺序是对应的。（3） item_completed(results, item, info) ，当一个item的所有下载请求都成功或失败时被调用。它的重写例子：from itemadapter import ItemAdapter from scrapy.exceptions import DropItem def item_completed(self, results, item, info): file_paths = [x['path'] for ok, x in results if ok] if not file_paths: raise DropItem("Item contains no files") adapter = ItemAdapter(item) adapter['file_paths'] = file_paths return item注意这个方法需要返回item，因为它要把下载的文件交给Media管道处理。Images管道类时Files管道类的扩展，修改了field名、添加了一些功能。（1） file_path(self, request, response=None, info=None, *, item=None) 方法，同FilesPipeline。（2） get_media_requests(item, info) ，和FilesPipeline相比只是更改了field名。（3） item_completed(results, item, info) ，和FilesPipeline相比只是更改了field名。6.AutoThrottle extension它的目的是通过scrapy负载和目标网页服务器的状态自动决定爬取速度。官方文档介绍了它的算法与原理，对于使用来说，你只需要知道在settings.py中设置 AUTOTHROTTLE_ENABLED = True 即可。更详细的设置如下：AUTOTHROTTLE_ENABLED¶ Default: False Enables the AutoThrottle extension. AUTOTHROTTLE_START_DELAY¶ Default: 5.0 The initial download delay (in seconds). AUTOTHROTTLE_MAX_DELAY¶ Default: 60.0 The maximum download delay (in seconds) to be set in case of high latencies. AUTOTHROTTLE_TARGET_CONCURRENCY¶ Default: 1.0 Average number of requests Scrapy should be sending in parallel to remote websites. By default, AutoThrottle adjusts the delay to send a single concurrent request to each of the remote websites. Set this option to a higher value (e.g. 2.0) to increase the throughput and the load on remote servers. A lower AUTOTHROTTLE_TARGET_CONCURRENCY value (e.g. 0.5) makes the crawler more conservative and polite. Note that CONCURRENT_REQUESTS_PER_DOMAIN and CONCURRENT_REQUESTS_PER_IP options are still respected when AutoThrottle extension is enabled. This means that if AUTOTHROTTLE_TARGET_CONCURRENCY is set to a value higher than CONCURRENT_REQUESTS_PER_DOMAIN or CONCURRENT_REQUESTS_PER_IP, the crawler won’t reach this number of concurrent requests. At every given time point Scrapy can be sending more or less concurrent requests than AUTOTHROTTLE_TARGET_CONCURRENCY; it is a suggested value the crawler tries to approach, not a hard limit. AUTOTHROTTLE_DEBUG¶ Default: False Enable AutoThrottle debug mode which will display stats on every response received, so you can see how the throttling parameters are being adjusted in real time.7.Benchmarking使用一个scrapy自带的最简爬虫测试你的硬件上限。简单的 scrapy bench 这一行命令即可，输出中会有几行： [scrapy.extensions.logstats] INFO: Crawled 518 pages (at 2880 pages/min), scraped 0 items (at 0 items/min) ，表示每分钟能爬2880个页面左右。但是从爬虫的复杂度、网页服务器的资源占用等角度考虑，实际环境中不会有这么快的速度。8.爬虫的暂停与恢复通过scheduler在硬盘中存储进度，去重器在硬盘上存储去重信息，加上能保存爬虫信息的一些扩展（extensions），可以做到暂停、恢复爬虫（中间可以关机的那种），这对大型网站的爬取尤其重要。通过在settings.py中指定 JOBDIR 就能做到这一点，但是要注意这个目录在不同的爬虫之间，或者同一个爬虫的不同实例之间都不能共享。比如你使用 scrapy crawl somespider -s JOBDIR=crawls/somespider-1 （不太清楚官方文档的JOBDIR为什么最后带一个 -1 ），然后对cmd或终端发送 ctrl+C 或其他一些信号时，就能安全地暂停爬虫；之后使用同样的 scrapy crawl 命令即可恢复爬虫。对于同一项目的多个爬虫，笔者认为可以在爬虫类中使用 custom_settings 这个属性指定 JOBDIR ，但是尚未实践。原文中给出的方法是使用爬虫的state属性：def parse_item(self, response): # parse item here self.state['items_count'] = self.state.get('items_count', 0) + 1最后需要提醒两点：（1）如果你的爬虫依赖Cookies，由于Cookies是会过期的，所以最好在暂停爬虫后不久就继续爬虫。另一个催促你尽快继续爬虫的因素是网页有可能会更新，导致你之前爬的数据有变化、过时或失效（比如爬取网页上提供的股市信息，而这个股市信息是每天更新的、只放出当天的）。（2）request对象也需要保存，这时会使用pickle模块，这就要求request对象是能用pickle序列化的。9.scrapy架构其实这个部分很尴尬，对初学者来说放在第一课感觉太枯燥，对老手来说没有必要讲。这里复制原文如下，图片自己随便找一张scrapy架构图看：This document describes the architecture of Scrapy and how its components interact. Overview¶ The following diagram shows an overview of the Scrapy architecture with its components and an outline of the data flow that takes place inside the system (shown by the red arrows). A brief description of the components is included below with links for more detailed information about them. The data flow is also described below. Data flow¶ Scrapy architecture The data flow in Scrapy is controlled by the execution engine, and goes like this: The Engine gets the initial Requests to crawl from the Spider. The Engine schedules the Requests in the Scheduler and asks for the next Requests to crawl. The Scheduler returns the next Requests to the Engine. The Engine sends the Requests to the Downloader, passing through the Downloader Middlewares (see process_request()). Once the page finishes downloading the Downloader generates a Response (with that page) and sends it to the Engine, passing through the Downloader Middlewares (see process_response()). The Engine receives the Response from the Downloader and sends it to the Spider for processing, passing through the Spider Middleware (see process_spider_input()). The Spider processes the Response and returns scraped items and new Requests (to follow) to the Engine, passing through the Spider Middleware (see process_spider_output()). The Engine sends processed items to Item Pipelines, then send processed Requests to the Scheduler and asks for possible next Requests to crawl. The process repeats (from step 3) until there are no more requests from the Scheduler. Components¶ Scrapy Engine¶ The engine is responsible for controlling the data flow between all components of the system, and triggering events when certain actions occur. See the Data Flow section above for more details. Scheduler¶ The scheduler receives requests from the engine and enqueues them for feeding them later (also to the engine) when the engine requests them. Downloader¶ The Downloader is responsible for fetching web pages and feeding them to the engine which, in turn, feeds them to the spiders. Spiders¶ Spiders are custom classes written by Scrapy users to parse responses and extract items from them or additional requests to follow. For more information see Spiders. Item Pipeline¶ The Item Pipeline is responsible for processing the items once they have been extracted (or scraped) by the spiders. Typical tasks include cleansing, validation and persistence (like storing the item in a database). For more information see Item Pipeline. Downloader middlewares¶ Downloader middlewares are specific hooks that sit between the Engine and the Downloader and process requests when they pass from the Engine to the Downloader, and responses that pass from Downloader to the Engine. Use a Downloader middleware if you need to do one of the following: process a request just before it is sent to the Downloader (i.e. right before Scrapy sends the request to the website); change received response before passing it to a spider; send a new Request instead of passing received response to a spider; pass response to a spider without fetching a web page; silently drop some requests. For more information see Downloader Middleware. Spider middlewares¶ Spider middlewares are specific hooks that sit between the Engine and the Spiders and are able to process spider input (responses) and output (items and requests). Use a Spider middleware if you need to post-process output of spider callbacks - change/add/remove requests or items; post-process start_requests; handle spider exceptions; call errback instead of callback for some of the requests based on response content. For more information see Spider Middleware. Event-driven networking¶ Scrapy is written with Twisted, a popular event-driven networking framework for Python. Thus, it’s implemented using a non-blocking (aka asynchronous) code for concurrency. For more information about asynchronous programming and Twisted see these links: Introduction to Deferreds Twisted - hello, asynchronous programming Twisted Introduction - Krondo10.下载中间件在settings.py中可以打开下载中间件，它的值越小表示离 process_request() 方法越早被唤醒、 process_response() 方法越晚被唤醒，或者理解为值越小离engine越近。这个值是比较有意义的，因为有时候你自己的下载中间件需要依赖之前的一些东西。默认的顺序如下：DOWNLOADER_MIDDLEWARES_BASE = { 'scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware': 100, 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware': 300, 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware': 350, 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware': 400, 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': 500, 'scrapy.downloadermiddlewares.retry.RetryMiddleware': 550, 'scrapy.downloadermiddlewares.ajaxcrawl.AjaxCrawlMiddleware': 560, 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware': 580, 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware': 590, 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware': 600, 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware': 700, 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': 750, 'scrapy.downloadermiddlewares.stats.DownloaderStats': 850, 'scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware': 900, }如果值设置为None，表示禁用这个中间件。如果要自己修改下载中间件，可以参考middlewares.py里的注释。另外，还有一些子类，如 scrapy.downloadermiddlewares.cookies.CookiesMiddleware ，用来保持session、存放cookies。你可以在settings.py中通过 COOKIES_ENABLED 和 COOKIES_DEBUG 来对它进行行为设置（默认前者打开，后者关闭）。默认情况下，一个项目中的爬虫使用同一cookies，你可以使用request对象的 meta 里的 cookiejar 键来修改这一行为（一个项目中的不同爬虫使用各自的cookies）：for i, url in enumerate(urls): yield scrapy.Request(url, meta={'cookiejar': i}, callback=self.parse_page)cookiejar这个键不太持久，每次在parse系列方法中生成request对象都要手动指定一次：def parse_page(self, response): # do some processing return scrapy.Request("http://www.example.com/otherpage", meta={'cookiejar': response.meta['cookiejar']}, callback=self.parse_other_page)另外，还有DefaultHeadersMiddleware、DownloadTimeoutMiddleware、HttpAuthMiddleware、HttpCacheMiddleware、HttpCompressionMiddleware、HttpProxyMiddleware、RedirectMiddleware和其他各种 DOWNLOADER_MIDDLEWARES_BASE 里出现的下载中间件，这里不介绍了。11.爬虫中间件具体见middlewares.py文件。12.添加extensions具体见原文：The extensions framework provides a mechanism for inserting your own custom functionality into Scrapy. Extensions are just regular classes. Extension settings¶ Extensions use the Scrapy settings to manage their settings, just like any other Scrapy code. It is customary for extensions to prefix their settings with their own name, to avoid collision with existing (and future) extensions. For example, a hypothetic extension to handle Google Sitemaps would use settings like GOOGLESITEMAP_ENABLED, GOOGLESITEMAP_DEPTH, and so on. Loading & activating extensions¶ Extensions are loaded and activated at startup by instantiating a single instance of the extension class per spider being run. All the extension initialization code must be performed in the class __init__ method. To make an extension available, add it to the EXTENSIONS setting in your Scrapy settings. In EXTENSIONS, each extension is represented by a string: the full Python path to the extension’s class name. For example: EXTENSIONS = { 'scrapy.extensions.corestats.CoreStats': 500, 'scrapy.extensions.telnet.TelnetConsole': 500, } As you can see, the EXTENSIONS setting is a dict where the keys are the extension paths, and their values are the orders, which define the extension loading order. The EXTENSIONS setting is merged with the EXTENSIONS_BASE setting defined in Scrapy (and not meant to be overridden) and then sorted by order to get the final sorted list of enabled extensions. As extensions typically do not depend on each other, their loading order is irrelevant in most cases. This is why the EXTENSIONS_BASE setting defines all extensions with the same order (0). However, this feature can be exploited if you need to add an extension which depends on other extensions already loaded. Available, enabled and disabled extensions¶ Not all available extensions will be enabled. Some of them usually depend on a particular setting. For example, the HTTP Cache extension is available by default but disabled unless the HTTPCACHE_ENABLED setting is set. Disabling an extension¶ In order to disable an extension that comes enabled by default (i.e. those included in the EXTENSIONS_BASE setting) you must set its order to None. For example: EXTENSIONS = { 'scrapy.extensions.corestats.CoreStats': None, } Writing your own extension¶ Each extension is a Python class. The main entry point for a Scrapy extension (this also includes middlewares and pipelines) is the from_crawler class method which receives a Crawler instance. Through the Crawler object you can access settings, signals, stats, and also control the crawling behaviour. Typically, extensions connect to signals and perform tasks triggered by them. Finally, if the from_crawler method raises the NotConfigured exception, the extension will be disabled. Otherwise, the extension will be enabled. Sample extension¶ Here we will implement a simple extension to illustrate the concepts described in the previous section. This extension will log a message every time: a spider is opened a spider is closed a specific number of items are scraped The extension will be enabled through the MYEXT_ENABLED setting and the number of items will be specified through the MYEXT_ITEMCOUNT setting. Here is the code of such extension: import logging from scrapy import signals from scrapy.exceptions import NotConfigured logger = logging.getLogger(__name__) class SpiderOpenCloseLogging: def __init__(self, item_count): self.item_count = item_count self.items_scraped = 0 @classmethod def from_crawler(cls, crawler): # first check if the extension should be enabled and raise # NotConfigured otherwise if not crawler.settings.getbool('MYEXT_ENABLED'): raise NotConfigured # get the number of items from settings item_count = crawler.settings.getint('MYEXT_ITEMCOUNT', 1000) # instantiate the extension object ext = cls(item_count) # connect the extension object to signals crawler.signals.connect(ext.spider_opened, signal=signals.spider_opened) crawler.signals.connect(ext.spider_closed, signal=signals.spider_closed) crawler.signals.connect(ext.item_scraped, signal=signals.item_scraped) # return the extension object return ext def spider_opened(self, spider): logger.info("opened spider %s", spider.name) def spider_closed(self, spider): logger.info("closed spider %s", spider.name) def item_scraped(self, item, spider): self.items_scraped += 1 if self.items_scraped % self.item_count == 0: logger.info("scraped %d items", self.items_scraped) Built-in extensions reference¶ General purpose extensions¶ Log Stats extension¶ classscrapy.extensions.logstats.LogStats[source]¶ Log basic stats like crawled pages and scraped items. Core Stats extension¶ classscrapy.extensions.corestats.CoreStats[source]¶ Enable the collection of core statistics, provided the stats collection is enabled (see Stats Collection). Telnet console extension¶ classscrapy.extensions.telnet.TelnetConsole[source]¶ Provides a telnet console for getting into a Python interpreter inside the currently running Scrapy process, which can be very useful for debugging. The telnet console must be enabled by the TELNETCONSOLE_ENABLED setting, and the server will listen in the port specified in TELNETCONSOLE_PORT. Memory usage extension¶ classscrapy.extensions.memusage.MemoryUsage[source]¶ Note This extension does not work in Windows. Monitors the memory used by the Scrapy process that runs the spider and: sends a notification e-mail when it exceeds a certain value closes the spider when it exceeds a certain value The notification e-mails can be triggered when a certain warning value is reached (MEMUSAGE_WARNING_MB) and when the maximum value is reached (MEMUSAGE_LIMIT_MB) which will also cause the spider to be closed and the Scrapy process to be terminated. This extension is enabled by the MEMUSAGE_ENABLED setting and can be configured with the following settings: MEMUSAGE_LIMIT_MB MEMUSAGE_WARNING_MB MEMUSAGE_NOTIFY_MAIL MEMUSAGE_CHECK_INTERVAL_SECONDS Memory debugger extension¶ classscrapy.extensions.memdebug.MemoryDebugger[source]¶ An extension for debugging memory usage. It collects information about: objects uncollected by the Python garbage collector objects left alive that shouldn’t. For more info, see Debugging memory leaks with trackref To enable this extension, turn on the MEMDEBUG_ENABLED setting. The info will be stored in the stats. Close spider extension¶ classscrapy.extensions.closespider.CloseSpider[source]¶ Closes a spider automatically when some conditions are met, using a specific closing reason for each condition. The conditions for closing a spider can be configured through the following settings: CLOSESPIDER_TIMEOUT CLOSESPIDER_ITEMCOUNT CLOSESPIDER_PAGECOUNT CLOSESPIDER_ERRORCOUNT Note When a certain closing condition is met, requests which are currently in the downloader queue (up to CONCURRENT_REQUESTS requests) are still processed. CLOSESPIDER_TIMEOUT¶ Default: 0 An integer which specifies a number of seconds. If the spider remains open for more than that number of second, it will be automatically closed with the reason closespider_timeout. If zero (or non set), spiders won’t be closed by timeout. CLOSESPIDER_ITEMCOUNT¶ Default: 0 An integer which specifies a number of items. If the spider scrapes more than that amount and those items are passed by the item pipeline, the spider will be closed with the reason closespider_itemcount. If zero (or non set), spiders won’t be closed by number of passed items. CLOSESPIDER_PAGECOUNT¶ Default: 0 An integer which specifies the maximum number of responses to crawl. If the spider crawls more than that, the spider will be closed with the reason closespider_pagecount. If zero (or non set), spiders won’t be closed by number of crawled responses. CLOSESPIDER_ERRORCOUNT¶ Default: 0 An integer which specifies the maximum number of errors to receive before closing the spider. If the spider generates more than that number of errors, it will be closed with the reason closespider_errorcount. If zero (or non set), spiders won’t be closed by number of errors. StatsMailer extension¶ classscrapy.extensions.statsmailer.StatsMailer[source]¶ This simple extension can be used to send a notification e-mail every time a domain has finished scraping, including the Scrapy stats collected. The email will be sent to all recipients specified in the STATSMAILER_RCPTS setting. Emails can be sent using the MailSender class. To see a full list of parameters, including examples on how to instantiate MailSender and use mail settings, see Sending e-mail. Debugging extensions¶ Stack trace dump extension¶ classscrapy.extensions.debug.StackTraceDump[source]¶ Dumps information about the running process when a SIGQUIT or SIGUSR2 signal is received. The information dumped is the following: engine status (using scrapy.utils.engine.get_engine_status()) live references (see Debugging memory leaks with trackref) stack trace of all threads After the stack trace and engine status is dumped, the Scrapy process continues running normally. This extension only works on POSIX-compliant platforms (i.e. not Windows), because the SIGQUIT and SIGUSR2 signals are not available on Windows. There are at least two ways to send Scrapy the SIGQUIT signal: By pressing Ctrl-while a Scrapy process is running (Linux only?) By running this command (assuming <pid> is the process id of the Scrapy process): kill -QUIT <pid> Debugger extension¶ classscrapy.extensions.debug.Debugger[source]¶ Invokes a Python debugger inside a running Scrapy process when a SIGUSR2 signal is received. After the debugger is exited, the Scrapy process continues running normally. For more info see Debugging in Python. This extension only works on POSIX-compliant platforms (i.e. not Windows).13.其他包括处理动态生成的数据（官方文档讲的很浅），处理内存泄漏，分布式部署爬虫（scrapyd），对Coroutines（协程）和asyncio（异步读写）的支持，各组件的API，scheduler，Item Exporters（导出为XML、CSV、JSON及JSON LINE等格式），这里就不介绍了。
本文较长，属于系统性总结，可作为使用手册查阅。步骤/目录： 1.scrapy介绍与安装 2.scrapy使用例一 3.scrapy命令行工具 4.爬虫类 （1）CrawlSpider （2）XMLFeedSpider与CSVFeedSpider （3）SitemapSpider 5.选择器对象 6.Items 7.Item Loaders 8.scrapy shell 9.Item Pipeline 10.Feed exports 11.request与response 12.link extractor 13.settings 14.Exceptions 15.logging 16.Stats Collection 17.Sending e-mail 18.Telnet Console 19.FAQ本文首发于个人博客https://lisper517.top/index.php/archives/51/，转载请注明出处。本文的目的是详细介绍scrapy，主要翻译自 scrapy官方文档 。本文写作日期为2022年9月14日。运行的平台为win10，编辑器为VS Code。1.scrapy介绍与安装scrapy（/ˈskreɪpaɪ/）是一个高效、通用的网页爬取框架，可用于提取页面数据、自动化测试网页、网页归档等。scrapy完全使用python语言（但是它依赖的库有些可能用了其他语言，比如C++），所以安装、使用scrapy之前需要安装、熟悉python。scrapy要求python版本在3.6以上，依赖的库有：lxml, an efficient XML and HTML parser parsel, an HTML/XML data extraction library written on top of lxml, w3lib, a multi-purpose helper for dealing with URLs and web page encodings twisted, an asynchronous networking framework cryptography and pyOpenSSL, to deal with various network-level security needs其中，有些依赖有版本要求：Twisted 14.0 lxml 3.4 pyOpenSSL 0.14使用 pip install Scrapy 即可安装scrapy及其依赖。如果用了Anaconda等包管理器，则使用 conda install -c conda-forge scrapy 命令安装scrapy。另外，官方文档最推荐的是在python虚拟环境中安装scrapy，以避免可能的包冲突，具体见 python官方文档 。2.scrapy使用例一安装好scrapy后，开始使用scrapy爬取一个网站： https://quotes.toscrape.com ，这是一个用来练习爬虫的网站。创建一个放scrapy爬虫的目录，比如 D:\spiders\scrapy 或 ~/spider/scrapy ，然后在cmd或终端中进入该目录，使用 scrapy startproject tutorial 来创建一个scrapy项目，这里的 tutorial 是项目名，你可以自行替换。然后在 tutorial/tutorial/spiders 目录下新建一个爬虫文件，比如 quotes_spider.py ，文件内容如下：import scrapy class QuotesSpider(scrapy.Spider): name = "quotes" def start_requests(self): urls = [ 'https://quotes.toscrape.com/page/1/', 'https://quotes.toscrape.com/page/2/', ] for url in urls: yield scrapy.Request(url=url, callback=self.parse) def parse(self, response): page = response.url.split("/")[-2] filename = f'quotes-{page}.html' with open(filename, 'wb') as f: f.write(response.body) self.log(f'Saved file {filename}')在这个爬虫类中， name 指定了爬虫名，在一个项目中的爬虫名不能相同； start_requests() 方法用于爬取初始网址，这个方法重写了父类方法，它需要返回可迭代的Requests对象； parse() 方法也重写了父类方法，它是默认的解析页面的方法，用于处理Response对象，Response对象是TextResponse类实例化来的，有一些帮助解析页面的属性和方法。 parse() 方法和其他自己写的 parse_xxx() 方法，它们的主要作用是解析响应对象，提取信息（以字典形式），找到页面上其他需要爬取的url、创建新的Request对象。接下来开启爬虫。在cmd或终端中进入项目文件夹，使用 scrapy crawl quotes 开启爬虫。注意这里跟的是爬虫名，不是爬虫文件名。之后，在cmd进入的目录下会生成两个html文件，即这次爬虫运行的结果。另外， start_requests() 方法也可以不写，而是改成写爬虫类的 start_urls 属性（起始url字符串的列表），这个列表中的url会自动被父类中的 start_requests() 方法调用，而不用显式指定：import scrapy class QuotesSpider(scrapy.Spider): name = "quotes" start_urls = [ 'https://quotes.toscrape.com/page/1/', 'https://quotes.toscrape.com/page/2/', ] def parse(self, response): page = response.url.split("/")[-2] filename = f'quotes-{page}.html' with open(filename, 'wb') as f: f.write(response.body)scrapy命令行还支持对Response进行调试，方法是 scrapy shell "url" 获取响应对象，然后使用 response.css('title') 、 response.css('title::text').re(r'Quotes.*') 、 response.xpath('//title') 等方法来查找html标签，但是这部分功能用浏览器自带的开发者模式（按F12）也能代替、且更方便，所以这里就不介绍了。最后，提一下Selector对象。 response.css() 方法和 response.xpath() 方法都返回选择器对象，而且在scrapy底层css选择器是转换成xpath表达式后查找标签的，而且xpath表达式比css选择器的功能更强，所以scrapy官方文档更推荐使用xpath，但是关于xpath表达式的内容这里就不展开了，有需要的可阅读笔者之前的文章 https://lisper517.top/index.php/archives/41/ 或参考 runoob教程 。Selector对象有 get() 、 getall() 方法来返回内容，不同在于前者只返回第一个、后者返回包含所有内容的列表。两个get方法是新方法，与之对应的旧方法是 extract_first() 、 extract() （现在它们的别名就是get和getall），但是现在新方法用的更多。它们之间只有一些微小的区别，get()只会返回一个结果，getall()总会返回一个列表（如果能取到东西的话），所以新方法的行为更加predictable。在刚才的爬虫中修改一些内容：import scrapy class QuotesSpider(scrapy.Spider): name = "quotes" start_urls = [ 'https://quotes.toscrape.com/page/1/', 'https://quotes.toscrape.com/page/2/', ] def parse(self, response): for quote in response.css('div.quote'): yield { 'text': quote.css('span.text::text').get(), 'author': quote.css('small.author::text').get(), 'tags': quote.css('div.tags a.tag::text').getall(), }python中的 yield 关键字功能和return类似，但用在递归程序中非常节约资源，在scrapy的爬虫中经常使用。再次用 scrapy crawl quotes 运行爬虫，会发现打印出了几个dict，这也是scrapy返回数据的格式。光在命令行中输出字典还不够，更常用的方法是把数据持久化存储，比如存成文件。在scrapy运行时就可以指定输出文件，如 scrapy crawl quotes -O quotes.jl ，虽然这种方式比较简陋，但命令行的 -O 参数会覆盖其他所有输出设置。这里的.jl文件比.json的好处是容易在后面附加数据（append）。以上提到的方法是在爬虫文件中就处理爬取到的数据，然而对于复杂一点的存储，更常用的方法是更改pipelines.py文件，即设置item pipeline，这部分内容会在后面讲解。scrapy更重要的功能是全站爬取，也就是对提取出来的url再次解析。在上面的例子中，可以把爬虫文件改写为：import scrapy class QuotesSpider(scrapy.Spider): name = "quotes" start_urls = [ 'https://quotes.toscrape.com/page/1/', ] def parse(self, response): for quote in response.css('div.quote'): yield { 'text': quote.css('span.text::text').get(), 'author': quote.css('small.author::text').get(), 'tags': quote.css('div.tags a.tag::text').getall(), } next_page = response.css('li.next a::attr(href)').get() if next_page is not None: next_page = response.urljoin(next_page) yield scrapy.Request(next_page, callback=self.parse)response.urljoin() 是用于有些时候，从a标签的href属性提取的网址不完整，这个方法会自动检测并补全网址。更重要的是这里用yield又构造了一个request对象， parse() 方法里出现了两次yield，这是yield和return使用的区别之一。在 scrapy.Request(next_page, callback=self.parse) 中，next_page指明下一个网址（scrapy将把这个网址放入请求队列，当下载器有空时就发出请求），callback指定调用的方法就是 parse() 方法自己，也就是说 parse() 方法自己调用了自己，这称为递归；有些时候callback也可以是自己写的其他 parse_xxx() 方法，这种情况一般是因为两个网页的格式不同。如果解析的是相近的页面，还可以使用 response.follow() 方法，这时可以省略 response.urljoin() 方法：import scrapy class QuotesSpider(scrapy.Spider): name = "quotes" start_urls = [ 'https://quotes.toscrape.com/page/1/', ] def parse(self, response): for quote in response.css('div.quote'): yield { 'text': quote.css('span.text::text').get(), 'author': quote.css('span small::text').get(), 'tags': quote.css('div.tags a.tag::text').getall(), } next_page = response.css('li.next a::attr(href)').get() if next_page is not None: yield response.follow(next_page, callback=self.parse)response.follow() 方法会自动调用urljoin。另外， response.follow(next_page, callback=self.parse) 的第一个参数还能换成Selector对象，只要这个标签有href属性；或者用 follow_all 方法，但是传入一个Selector对象的列表甚至是css表达式，格式稍微改一下：import scrapy class QuotesSpider(scrapy.Spider): name = "quotes" start_urls = [ 'https://quotes.toscrape.com/page/1/', ] def parse(self, response): for quote in response.css('div.quote'): yield { 'text': quote.css('span.text::text').get(), 'author': quote.css('span small::text').get(), 'tags': quote.css('div.tags a.tag::text').getall(), } yield from response.follow_all(css='ul.pager a', callback=self.parse)正如前面提到的，callback指定的回调函数可以是其他自己写的 parse_xxx() 方法：import scrapy class AuthorSpider(scrapy.Spider): name = 'author' start_urls = ['https://quotes.toscrape.com/'] def parse(self, response): author_page_links = response.css('.author + a') yield from response.follow_all(author_page_links, self.parse_author) pagination_links = response.css('li.next a') yield from response.follow_all(pagination_links, self.parse) def parse_author(self, response): def extract_with_css(query): return response.css(query).get(default='').strip() yield { 'name': extract_with_css('h3.author-title::text'), 'birthdate': extract_with_css('.author-born-date::text'), 'bio': extract_with_css('.author-description::text'), }以上所有的方法都不需要担心重复访问同一个网址，因为scrapy自带网址去重。要关掉这个功能，可以在follow系列方法中使用 dont_filter=True ，或者设置DUPEFILTER_CLASS类来指定，但不推荐这样做。在 parse 系列方法中还可以传递item数据，称为请求传参，这将在后续介绍。现在稍微提一下使用场景，比如你要爬取一个博客，博客的目录页有博文的标题、上传时间，但是又要进入每篇博文爬取详细内容，这时就可以使用请求传参。3.scrapy命令行工具scrapy在命令行中使用时有一些注意事项。首先，在项目目录里有一个scrapy.cfg文件，它是这个项目的配置文件，优先级最高（相对于scrapy的全局配置、用户配置）。在这个文件中，可以指定多个项目共用一个settings.py文件。在cmd输入 scrapy ，如果cmd在scrapy项目目录之内，第一行会显示当前项目内是否有活动的爬虫，其他行则是使用帮助。下面就详细讲解scrapy命令行， [] 括起来表示可选项。。（1）查看某个命令的详细帮助： scrapy <command> -h 。（2）创建项目： scrapy startproject 项目名 [项目目录名] 。创建项目后进入项目目录。（3）创建爬虫： scrapy genspider [-t template] <爬虫名> <待爬域名> ， -t 指定爬虫模板，待爬域名可以随便写。（4）查看爬虫模板： scrapy genspider -l 。（5）开启爬虫： scrapy crawl <spider> 。（6）检测爬虫： scrapy check [-l] <spider> 。（7）输出项目中所有爬虫： scrapy list 。（8）编辑爬虫： scrapy edit <spider> ，将用默认的编辑器打开并编辑爬虫，这个编辑器可用 EDITOR 环境变量指定。（9）爬取一个网页： scrapy fetch <url> ，爬取网页并输出其内容到standard output。在项目中使用时，它是根据settings.py或爬虫的设置取网页的，比如有时你在settings.py里设置了UA，或者对爬虫有一些个性化设置。可选项有 --spider=SPIDER （使用某个爬虫的设置）、 --headers （打印请求头）、 --no-redirect （不重定向）。（10）查看网页： scrapy view <url> ，可选项有 --spider=SPIDER 、 --no-redirect 。和 fetch 一起，这两个命令可用于检查爬虫和自己看到的网页有无不同。（11）查看设置： scrapy settings [options] ，比如 scrapy settings --get BOT_NAME 。在项目内时这个命令显示项目设置，否则显示默认设置。（12）运行单独爬虫文件： scrapy runspider <spider_file.py> ，运行一个能独立运行的爬虫文件，很少用。（13）显示scrapy版本： scrapy version [-v] 。其他命令用的不多，这里就不介绍了。另外，可以自己定制scrapy命令，需要用到 COMMANDS_MODULE 这项设置，具体参考原文。4.爬虫类在 scrapy使用例一中 介绍最多的就是爬虫文件中的爬虫类。所有scrapy爬虫类都需要继承自scrapy.Spider父类，这个父类的主要功能有 start_requests() 、 start_urls 、 parse() ，这些已经在前文介绍过了。其他还有一些属性及方法：（1）name，爬虫名字，用于区别项目内的各个爬虫。一般来说，一个爬虫类只会实例化一个对象，但也能实例化多个。常用的name编写方法是根据爬取的域名来截取。（2）allowed_domains，允许爬取的域名（包括子域名），这个一般可以注释掉。（3）start_urls，前文提过。（4）custom_settings，用于指定爬虫的个性化设置，会覆盖掉settings.py里的设置。注意它必须写在类属性而不能在__init__里指定，因为实例化之前就会上传设置。（5）logger，用爬虫名创建的logger对象，可以自己写log信息。或者也可用 log(message[, level, component]) 方法写log。（6）closed(reason)，当爬虫结束时调用一次这个方法。（7）start_requests() 和 parse(response) ，前文已介绍。另外，scrapy提供了一些衍生的爬虫类，方便爬取全站数据，或者根据sitemap爬取，或者解析XML feed。下面的例子中，假设 items.py 文件如下：import scrapy class TestItem(scrapy.Item): id = scrapy.Field() name = scrapy.Field() description = scrapy.Field()（1）CrawlSpider即scrapy.spiders.CrawlSpider，常用于爬取全站数据，它的特点是提取规则（rules）。一个CrawlSpider例子如下：import scrapy from scrapy.linkextractors import LinkExtractor from scrapy.spiders import CrawlSpider, Rule class CrawlspiSpider(CrawlSpider): name = 'crawlspi' #allowed_domains = ['xxx.com'] start_urls = ['http://www.xxx.com/'] rules = ( Rule(LinkExtractor(allow=r'page=\d+'), callback='parse_item', follow=True), Rule(LinkExtractor(allow=r'xxxx'), callback='parse_detail', follow=True), Rule(LinkExtractor(restrict_xpaths='//a', allow=r'/wzshow_\d+\.html'), callback='parse_item'), # 还可以通过xpath解析网页 ) # LinkExtractor(allow=r'Items/')根据规则提取链接（正则） # Rule对象会对提取到的网页自动发送请求并通过callback解析（无法使用请求传参） # follow=True即对提取到的链接页面再应用规则提取链接（全站爬取） # 一般第一个Rule提取其他页码链接，第二个Rule提取当前页面详情url # 对请求传参，在items中再写一个DetailItem类，并且有一个共同的主键PRIMARY KEY用于识别 # 在Pipeline中对item.__class__.__name__判断，对不同的item类处理 # 插入Mysql中时通过主键处理 def parse_item(self, response): item = {} #item['domain_id'] = response.xpath('//input[@id="sid"]/@value').get() #item['name'] = response.xpath('//div[@id="name"]').get() #item['description'] = response.xpath('//div[@id="description"]').get() return item def parse_detail(self, response): item = {} return item如果创建了多个Rule对象，而一个链接符合多个规则，则会应用第一个匹配的规则。提取规则对象的完整实例化格式如下：scrapy.spiders.Rule(link_extractor=None, callback=None, cb_kwargs=None, follow=None, process_links=None, process_request=None, errback=None)link_extractor指定Link Extractor对象，如果省略则匹配所有提取的链接。它是用来生成request对象的，这里的request对象的 meta['link_text'] 可以给出请求的url；callback是一个callable方法或字符串，用于调用指定的方法，一般是parse系列方法；cb_kwargs是一个参数字典，将传递给回调函数（所以也可以用请求传参，上面的注释适用于1.7版本之前的scrapy）；follow则指定，对于使用规则提取到的网页，是否继续应用规则提取链接；process_links，也是一个callable方法或字符串，用于调用指定的方法，但这个方法主要用来对提取到的链接进行筛选；process_request，也是一个callable方法或字符串，用于调用指定的方法（该方法的第一个参数为request对象，第二个参数为该request对象生成的response对象，返回一个request对象或者None），它主要对请求进行筛选（返回None时就是筛选掉了请求对象）；errback是发生exception时的回调函数。官方文档给出的例子是：import scrapy from scrapy.spiders import CrawlSpider, Rule from scrapy.linkextractors import LinkExtractor class MySpider(CrawlSpider): name = 'example.com' allowed_domains = ['example.com'] start_urls = ['http://www.example.com'] rules = ( # Extract links matching 'category.php' (but not matching 'subsection.php') # and follow links from them (since no callback means follow=True by default). Rule(LinkExtractor(allow=('category\.php', ), deny=('subsection\.php', ))), # Extract links matching 'item.php' and parse them with the spider's method parse_item Rule(LinkExtractor(allow=('item\.php', )), callback='parse_item'), ) def parse_item(self, response): self.logger.info('Hi, this is an item page! %s', response.url) item = scrapy.Item() item['id'] = response.xpath('//td[@id="item_id"]/text()').re(r'ID: (\d+)') item['name'] = response.xpath('//td[@id="item_name"]/text()').get() item['description'] = response.xpath('//td[@id="item_description"]/text()').get() item['link_text'] = response.meta['link_text'] url = response.xpath('//td[@id="additional_data"]/@href').get() return response.follow(url, self.parse_additional_page, cb_kwargs=dict(item=item)) def parse_additional_page(self, response, item): item['additional_data'] = response.xpath('//p[@id="additional_data"]/text()').get() return item注意这里用的都是return而非yield。（2）XMLFeedSpider与CSVFeedSpider分别用于解析xml的feed网页和csv，这里就不详细介绍了。它们也使用return而非yield。（3）SitemapSpider根据sitemap爬网页，这个其实用的也很少。它能手动指定sitemap网页，或者根据robots.txt找到sitemap网页。5.选择器对象在爬虫界，常用的解析网页html或xml的方式有beautifulsoup（缺点是慢）、正则表达式（麻烦，容易解析错误）、lxml；lxml又分为css、xpath等。scrapy的Selector对象提供了一些方法，也能解析html，它在底层使用的是lxml的xpath表达式，解析速度也差不多。正如前文提到的， response.css() 和 response.xpath() 方法返回选择器对象列表，对选择器对象列表可使用 get() 和 getall() 方法，这两个方法如果取不到对象就会返回None，但是如果用 get(default="some-string") 也可以指定取不到时返回什么。除了这两个方法，scrapy还提供了选择器的其他方法来取出属性等，比如attrib、re方法，这里就不介绍了。另外，在xpath函数里使用 .//text() 时最好用 . 代替；xpath还可以传递参数，比如 response.xpath('//div[count(a)=$cnt]/@id', cnt=5) 。scrapy还为xpath提供了自己的has-class方法： response.xpath('//p[has-class("foo", "bar-baz")]') ，这样就能选中像 First 和 Second 这样的标签（属性值变化），但它比css方法稍慢。parsel.xpathfuncs.set_xpathfunc(fname, func) 可以自己注册xpath函数。选择器对象的构造函数为 scrapy.selector.Selector(*args, **kwargs) ，参数有： response ，指定HtmlResponse或XmlResponse对象； text ，指定响应的正文，不能和 response 一起用； type ，指定格式，可选择html、xml或None，为None时自动选择格式；另外，为了方便，scrapy还自带了选择器列表的类，它衍生于基础的list，这样对于元素为选择器的列表也能使用以上方法。6.Itemsitems系列的类就是字典或者可以当成字典使用，用于爬虫处理完信息后的传递。items类在items.py中进行定义，常见的有：（1）字典，就是python字典。（2）item对象，它提供了类似字典的API和一些其他功能，构造函数为 scrapy.item.Item([arg]) 或 scrapy.Item([arg]) ，简单来说你需要在items.py里规定一个field，然后才能在爬虫中使用这个field，否则会产生exception。另外，还有 Item.copy() 和 Item.deepcopy() 方法用于浅拷贝和深拷贝item对象。一个使用item对象的items.py文件如下：import scrapy class IpoolItem(scrapy.Item): one_field = scrapy.Field() another_field = scrapy.Field()（3）dataclass对象，它支持对每个field设置默认值，使用例为：from dataclasses import dataclass @dataclass class CustomItem: one_field: str another_field: int虽然规定了field的类型，但在爬虫中为field赋值时不会检查类型。（4）attr.s对象，使用例为：import attr @attr.s class CustomItem: one_field = attr.ib() another_field = attr.ib()同样支持默认值。items系列的对象可以使用字典的 keys() 、 items() 方法。另外，由于python的机制，在items中存放列表或字典等数据时，如果要拷贝则应当使用深拷贝。最后，items对象还支持一些metadata（超参数）；你可以从自己定义的item类衍生子类，来增加field或修改field的超参数：class IpoolItem(scrapy.Item): one_field = scrapy.Field() another_field = scrapy.Field() class IpoolItem(IpoolItem): third_field = scrapy.Field()7.Item Loadersitem提供了存放数据的容器，而item loader可以帮助你更便捷地把数据放进容器。比方说，你可以使用item loader多次对一个field赋值，item loader会把你赋的值组合起来（字符串会append）。item loader是在爬虫文件中使用的，使用例：from scrapy.loader import ItemLoader from myproject.items import Product def parse(self, response): l = ItemLoader(item=ProductItem(), response=response) l.add_xpath('name', '//div[@class="product_name"]') l.add_xpath('name', '//div[@class="product_title"]') l.add_xpath('price', '//p[@id="price"]') l.add_css('stock', 'p#stock') l.add_value('last_updated', 'today') # you can also use literal values return l.load_item()这里使用了 add_xpath() 、 add_css() 、 add_value() 、 load_item() 方法，注意它们的行为。在使用 load_item() 之前，所有的数据都暂存在item loader里，使用之后才会赋值到item的field中。实际上，对于每个field，每次在item loader中赋值时都是从一个input processor进入，使用 load_item() 时从一个output processor出来，这涉及到一些其他用法。Itemloader的构造函数为 scrapy.loader.ItemLoader(item=None, selector=None, response=None, parent=None, **context) ，其中 selector 和 response 是互斥的，前者优先。另外，如果要更改值而不是append，ItemLoader类提供了replace系列方法，就是把add系列中的add换成replace。最后，Itemloader可以停留在某个节点位置：loader = ItemLoader(item=Item()) # load stuff not in the footer footer_loader = loader.nested_xpath('//footer') footer_loader.add_xpath('social', 'a[@class = "social"]/@href') footer_loader.add_xpath('email', 'a[@class = "email"]/@href') # no need to call footer_loader.load_item() loader.load_item()这里用的 add_xpath 中的xpath表达式都是相对于footer标签的位置，也就是说完整的xpath表达式为 //footer/a[@class = "social"]/@href 。总的来说，itemloader是为了简化item生成而存在的，在某些情况下有用，但一般不会用到。8.scrapy shell它的主要作用是检测xpath、css表达式。由于浏览器的开发者模式能更便捷地做到相同的事情，这里就不介绍了。9.Item Pipelineitem管道，在爬虫生成item类后会把这些item数据传递到管道中进行处理，即scrapy项目中的pipelines.py文件。那你可能就要问为什么我不在爬虫中，刚爬到数据就把数据处理了呢？这一方面是为了代码的简洁性，另一方面管道有一些独到的优势。一般来说，管道可以用于：理清html数据，验证数据，数据去重，数据存入数据库。在pipelines.py中可以写你自己的管道，必须提供的方法只有 process_item(self, item, spider) ，它用于处理item，它必须返回一个item对象或Twisted的Deferred，或者raise一个Dropitem的exception（此时这个item就不会向下面的管道继续传递）。另外，你还可以选择写以下方法：open_spider(self, spider) ，只在爬虫开启时调用一次，你可以在这里使用pymysql创建connection、cursor，或者创建一个文件读写对象；相对的是 close_spider(self, spider) ，只在爬虫关闭时调用一次，可以关闭conn、cur、文件读写对象；另外还有 from_crawler(cls, crawler) 方法，当它存在时，会根据crawler对象实例化一个管道，所以它也必须返回一个管道对象，这个方法的作用是为管道提供访问settings等核心的途径，以及提供函数钩子（hook）。管道文件的范例：from itemadapter import ItemAdapter from scrapy.exceptions import DropItem class PricePipeline: vat_factor = 1.15 def process_item(self, item, spider): adapter = ItemAdapter(item) if adapter.get('price'): if adapter.get('price_excludes_vat'): adapter['price'] = adapter['price'] * self.vat_factor return item else: raise DropItem(f"Missing price in {item}")再来一个（存入mongodb）：import pymongo from itemadapter import ItemAdapter class MongoPipeline: collection_name = 'scrapy_items' def __init__(self, mongo_uri, mongo_db): self.mongo_uri = mongo_uri self.mongo_db = mongo_db @classmethod def from_crawler(cls, crawler): return cls( mongo_uri=crawler.settings.get('MONGO_URI'), mongo_db=crawler.settings.get('MONGO_DATABASE', 'items') ) def open_spider(self, spider): self.client = pymongo.MongoClient(self.mongo_uri) self.db = self.client[self.mongo_db] def close_spider(self, spider): self.client.close() def process_item(self, item, spider): self.db[self.collection_name].insert_one(ItemAdapter(item).asdict()) return item最后，如果自己在pipelines.py里写了其他管道，或者修改了默认管道名，需要在settings.py里打开管道：ITEM_PIPELINES = { 'myproject.pipelines.管道名1': 300, 'myproject.pipelines.管道名2': 800, }后面的300、800，它们的取值范围为0-1000，item将从低值的管道流经高值的管道。10.Feed exports你可以用Feed exports来把数据导出成json、json lines（jl）、csv或xml等格式，或者python的pickle，这样数据就能用于不同的系统了。这个功能主要需要在settings.py中开启，相关的设置有：FEEDS FEED_EXPORT_ENCODING FEED_STORE_EMPTY FEED_EXPORT_FIELDS FEED_EXPORT_INDENT FEED_STORAGES FEED_STORAGE_FTP_ACTIVE FEED_STORAGE_S3_ACL FEED_EXPORTERS FEED_EXPORT_BATCH_ITEM_COUNT第一项是必须的。范例：FEEDS={ 'items.json': { 'format': 'json', 'encoding': 'utf8', 'store_empty': False, 'item_classes': [MyItemClass1, 'myproject.items.MyItemClass2'], 'fields': None, 'indent': 4, 'item_export_kwargs': { 'export_empty_fields': True, }, }, '/home/user/documents/items.xml': { 'format': 'xml', 'fields': ['name', 'price'], 'item_filter': MyCustomFilter1, 'encoding': 'latin1', 'indent': 8, }, pathlib.Path('items.csv.gz'): { 'format': 'csv', 'fields': ['price', 'name'], 'item_filter': 'myproject.filters.MyCustomFilter2', 'postprocessing': [MyPlugin1, 'scrapy.extensions.postprocessing.GzipPlugin'], 'gzip_compresslevel': 5, }, }各个参数的解释：The following is a list of the accepted keys and the setting that is used as a fallback value if that key is not provided for a specific feed definition: format: the serialization format. This setting is mandatory, there is no fallback value. batch_item_count: falls back to FEED_EXPORT_BATCH_ITEM_COUNT. New in version 2.3.0. encoding: falls back to FEED_EXPORT_ENCODING. fields: falls back to FEED_EXPORT_FIELDS. item_classes: list of item classes to export. If undefined or empty, all items are exported. New in version 2.6.0. item_filter: a filter class to filter items to export. ItemFilter is used be default. New in version 2.6.0. indent: falls back to FEED_EXPORT_INDENT. item_export_kwargs: dict with keyword arguments for the corresponding item exporter class. New in version 2.4.0. overwrite: whether to overwrite the file if it already exists (True) or append to its content (False). The default value depends on the storage backend: Local filesystem: False FTP: True Note Some FTP servers may not support appending to files (the APPE FTP command). S3: True (appending is not supported) Standard output: False (overwriting is not supported) New in version 2.4.0. store_empty: falls back to FEED_STORE_EMPTY. uri_params: falls back to FEED_URI_PARAMS. postprocessing: list of plugins to use for post-processing. The plugins will be used in the order of the list passed. New in version 2.6.0.其他设置的解析：FEED_EXPORT_ENCODING¶ Default: None The encoding to be used for the feed. If unset or set to None (default) it uses UTF-8 for everything except JSON output, which uses safe numeric encoding (\uXXXX sequences) for historic reasons. Use utf-8 if you want UTF-8 for JSON too. FEED_EXPORT_FIELDS¶ Default: None A list of fields to export, optional. Example: FEED_EXPORT_FIELDS = ["foo", "bar", "baz"]. Use FEED_EXPORT_FIELDS option to define fields to export and their order. When FEED_EXPORT_FIELDS is empty or None (default), Scrapy uses the fields defined in item objects yielded by your spider. If an exporter requires a fixed set of fields (this is the case for CSV export format) and FEED_EXPORT_FIELDS is empty or None, then Scrapy tries to infer field names from the exported data - currently it uses field names from the first item. FEED_EXPORT_INDENT¶ Default: 0 Amount of spaces used to indent the output on each level. If FEED_EXPORT_INDENT is a non-negative integer, then array elements and object members will be pretty-printed with that indent level. An indent level of 0 (the default), or negative, will put each item on a new line. None selects the most compact representation. Currently implemented only by JsonItemExporter and XmlItemExporter, i.e. when you are exporting to .json or .xml. FEED_STORE_EMPTY¶ Default: False Whether to export empty feeds (i.e. feeds with no items). FEED_STORAGES¶ Default: {} A dict containing additional feed storage backends supported by your project. The keys are URI schemes and the values are paths to storage classes. FEED_STORAGE_FTP_ACTIVE¶ Default: False Whether to use the active connection mode when exporting feeds to an FTP server (True) or use the passive connection mode instead (False, default). For information about FTP connection modes, see What is the difference between active and passive FTP?. FEED_STORAGE_S3_ACL¶ Default: '' (empty string) A string containing a custom ACL for feeds exported to Amazon S3 by your project. For a complete list of available values, access the Canned ACL section on Amazon S3 docs. FEED_STORAGES_BASE¶ Default: { '': 'scrapy.extensions.feedexport.FileFeedStorage', 'file': 'scrapy.extensions.feedexport.FileFeedStorage', 'stdout': 'scrapy.extensions.feedexport.StdoutFeedStorage', 's3': 'scrapy.extensions.feedexport.S3FeedStorage', 'ftp': 'scrapy.extensions.feedexport.FTPFeedStorage', } A dict containing the built-in feed storage backends supported by Scrapy. You can disable any of these backends by assigning None to their URI scheme in FEED_STORAGES. E.g., to disable the built-in FTP storage backend (without replacement), place this in your settings.py: FEED_STORAGES = { 'ftp': None, } FEED_EXPORTERS¶ Default: {} A dict containing additional exporters supported by your project. The keys are serialization formats and the values are paths to Item exporter classes. FEED_EXPORTERS_BASE¶ Default: { 'json': 'scrapy.exporters.JsonItemExporter', 'jsonlines': 'scrapy.exporters.JsonLinesItemExporter', 'jl': 'scrapy.exporters.JsonLinesItemExporter', 'csv': 'scrapy.exporters.CsvItemExporter', 'xml': 'scrapy.exporters.XmlItemExporter', 'marshal': 'scrapy.exporters.MarshalItemExporter', 'pickle': 'scrapy.exporters.PickleItemExporter', } A dict containing the built-in feed exporters supported by Scrapy. You can disable any of these exporters by assigning None to their serialization format in FEED_EXPORTERS. E.g., to disable the built-in CSV exporter (without replacement), place this in your settings.py: FEED_EXPORTERS = { 'csv': None, } FEED_EXPORT_BATCH_ITEM_COUNT¶ New in version 2.3.0. Default: 0 If assigned an integer number higher than 0, Scrapy generates multiple output files storing up to the specified number of items in each output file. When generating multiple output files, you must use at least one of the following placeholders in the feed URI to indicate how the different output file names are generated: %(batch_time)s - gets replaced by a timestamp when the feed is being created (e.g. 2020-03-28T14-45-08.237134) %(batch_id)d - gets replaced by the 1-based sequence number of the batch. Use printf-style string formatting to alter the number format. For example, to make the batch ID a 5-digit number by introducing leading zeroes as needed, use %(batch_id)05d (e.g. 3 becomes 00003, 123 becomes 00123). For instance, if your settings include: FEED_EXPORT_BATCH_ITEM_COUNT = 100 And your crawl command line is: scrapy crawl spidername -o "dirname/%(batch_id)d-filename%(batch_time)s.json" The command line above can generate a directory tree like: ->projectname -->dirname --->1-filename2020-03-28T14-45-08.237134.json --->2-filename2020-03-28T14-45-09.148903.json --->3-filename2020-03-28T14-45-10.046092.json Where the first and second files contain exactly 100 items. The last one contains 100 items or fewer. FEED_URI_PARAMS¶ Default: None A string with the import path of a function to set the parameters to apply with printf-style string formatting to the feed URI. The function signature should be as follows: scrapy.extensions.feedexport.uri_params(params, spider)¶ Return a dict of key-value pairs to apply to the feed URI using printf-style string formatting. Parameters params (dict) – default key-value pairs Specifically: batch_id: ID of the file batch. See FEED_EXPORT_BATCH_ITEM_COUNT. If FEED_EXPORT_BATCH_ITEM_COUNT is 0, batch_id is always 1. New in version 2.3.0. batch_time: UTC date and time, in ISO format with : replaced with -. See FEED_EXPORT_BATCH_ITEM_COUNT. New in version 2.3.0. time: batch_time, with microseconds set to 0. spider (scrapy.Spider) – source spider of the feed items Caution The function should return a new dictionary, modifying the received params in-place is deprecated. For example, to include the name of the source spider in the feed URI: Define the following function somewhere in your project: # myproject/utils.py def uri_params(params, spider): return {**params, 'spider_name': spider.name} Point FEED_URI_PARAMS to that function in your settings: # myproject/settings.py FEED_URI_PARAMS = 'myproject.utils.uri_params' Use %(spider_name)s in your feed URI: scrapy crawl <spider_name> -o "%(spider_name)s.jl"因为笔者用这个功能不多，这里就不翻译了。11.request与responserequest对象由爬虫生成，最后传递到下载器；response则是由下载器根据request生成，返回给发出request的爬虫。二者都有一些衍生类，下面先介绍一下基类。classscrapy.http.Request(*args, **kwargs) ，它的参数有： （1）url ，一个网址字符串；.（2）callback ，当request返回response后会用这个参数指定的回调函数接受response对象，所以这个回调函数一般是爬虫中 parse 系列的方法，默认是用 parse() ，而解析出错时会调用 errback 指定的回调函数。（3） method， 字符串，发起请求的方法，默认是 GET .（4） meta ，字典，它会赋给request.meta数据成员（浅拷贝），就是之前提到的请求传参（但是现在更推荐使用 cb_kwargs 来请求传参）。它还会传递给对应的response对象。（5） body ，可能是bytes或str，表示请求体，实际的请求体会以二进制形式存储。（6） headers ，字典，就是请求头。（7） cookies ，字典，用于设置请求头里的cookie，但目前版本的scrapy（2.6）这个参数还不能使用，需要设置cookie的话用 request.cookies 在下载中间件或CookiesMiddlewares里设置，另外需要注意的是cookies会根据网站返回的cookies更改，这点更像真的浏览器，要关闭这个功能可以在 request.meta 里把 dont_merge_cookies 赋值为True。（8） encoding ，字符串，指定request的字符集，默认为 uft-8 。这个编码格式会用来对url进行百分号编码（如果平时观察过url，会发现空格转换成了%20，汉字也用%转码），并且将请求体转换成二进制。（9） priority ，这个请求的优先级，默认为0。scheduler将用优先级来决定请求发送的顺序，越大的优先级越高。（10） dont_filter ，scheduler会对网址进行去重过滤，这个参数为True时就不会去重过滤。比如有些网址每次访问的结果都不同，就可以把去重关闭，但要小心使用、避免loop。（11） errback ，处理request出现exception时会调用的函数，包括页面的404错误。（12） flags ，list，用来logging。（13） cb_kwargs ，字典，就是callback指定的回调函数的参数字典。下面介绍request对象的成员：（1） url ，str，请求网址。注意这个网址是被百分号编码过的。它是read-only的，要替换它请使用request对象的 replace() 方法，见后。（2） method ，str，表示请求方法，比如 "GET" ， "POST" ， "PUT" 。（3） headers ，类似字典的对象，表示请求头。（4） body ，二进制的请求体。同样是只读的，可以用 replace() 方法更改。（5） meta ，字典，用来存储一些metadata，可以被其他extensions、中间件访问。另外，request的meta字典会传递给对应的response。（6） cb_kwargs ，字典，传递给回调函数的参数字典。（7） attributes: Tuple[str, ...]= ('url', 'callback', 'method', 'headers', 'body', 'cookies', 'meta', 'encoding', 'priority', 'dont_filter', 'errback', 'flags', 'cb_kwargs') ，一个包含请求的基本信息的字符串元组。被 Request.replace() ， Request.to_dict() 和 request_from_dict() 调用。（8） copy()，返回一个请求对象的浅拷贝。（9） replace([url, method, headers, body, cookies, meta, flags, encoding, priority, dont_filter, callback, errback, cb_kwargs]) ，它会返回一个请求对象，所有提到的请求对象成员会被替换为新值、未提到的保留原值，用于更改请求对象的一些成员。这里也是浅拷贝。（10） to_dict(*, spider: Optional[Spider] = None)→ dict ，和 attributes 方法功能类似，但是返回一个字典。使用这个字典，搭配 scrapy.utils.request.request_from_dict(d: dict, *, spider: Optional[Spider] = None)→ Request 方法，可以根据字典生成request对象。 spider: Optional[Spider] = None 可选项用于根据爬虫名找到对应的爬虫，也就是说这个request生成的response对象必须用该爬虫的 parse 方法。下面介绍一下errback回调函数的书写格式。它接受的第一个参数必须是failure，使用例：import scrapy from scrapy.spidermiddlewares.httperror import HttpError from twisted.internet.error import DNSLookupError from twisted.internet.error import TimeoutError, TCPTimedOutError class ErrbackSpider(scrapy.Spider): name = "errback_example" start_urls = [ "http://www.httpbin.org/", # HTTP 200 expected "http://www.httpbin.org/status/404", # Not found error "http://www.httpbin.org/status/500", # server issue "http://www.httpbin.org:12345/", # non-responding host, timeout expected "https://example.invalid/", # DNS error expected ] def start_requests(self): for u in self.start_urls: yield scrapy.Request(u, callback=self.parse_httpbin, errback=self.errback_httpbin, dont_filter=True) def parse_httpbin(self, response): self.logger.info('Got successful response from {}'.format(response.url)) # do something useful here... def errback_httpbin(self, failure): # log all failures self.logger.error(repr(failure)) # in case you want to do something special for some errors, # you may need the failure's type: if failure.check(HttpError): # these exceptions come from HttpError spider middleware # you can get the non-200 response response = failure.value.response self.logger.error('HttpError on %s', response.url) elif failure.check(DNSLookupError): # this is the original request request = failure.request self.logger.error('DNSLookupError on %s', request.url) elif failure.check(TimeoutError, TCPTimedOutError): request = failure.request self.logger.error('TimeoutError on %s', request.url)以及用failure.request.cb_kwargs的例子：def parse(self, response): request = scrapy.Request('http://www.example.com/index.html', callback=self.parse_page2, errback=self.errback_page2, cb_kwargs=dict(main_url=response.url)) yield request def parse_page2(self, response, main_url): pass def errback_page2(self, failure): yield dict( main_url=failure.request.cb_kwargs['main_url'], )下面介绍一下meta这个超数据。2.6版本的scrapy，一般不用它来请求传参，而是用于在各组件之间传递信息。它自带的键有：bindaddress cookiejar dont_cache dont_merge_cookies dont_obey_robotstxt dont_redirect dont_retry download_fail_on_dataloss download_latency download_maxsize download_timeout ftp_password (See FTP_PASSWORD for more info) ftp_user (See FTP_USER for more info) handle_httpstatus_all handle_httpstatus_list max_retry_times proxy redirect_reasons redirect_urls referrer_policy比较值得注意的可能是proxy，可以在下载中间件中为请求对象设置代理。另外scrapy还提供了一些request对象的子类，包括：（1）FormRequest，主要用于处理html的表单，比如用于模拟登录（简单的输入用户名密码），但是这个功能使用request对象也不复杂。（2）JsonRequest，用于处理会返回json数据的请求，也没什么用。最后介绍一下response对象。构造函数为 classscrapy.http.Response(*args, **kwargs) ，参数有：（1） url ，字符串，响应来源网址（原文未提到是否被百分号转义过，似乎没有；原文提到Response.request.url doesn’t always equal Response.url）。（2） status ，int，响应的状态码。这和http协议有关，正常情况下一般是200，如果是404等就表示错误。（3） headers ，字典，响应头。（4） body ，bytes，二进制的响应体。如果希望拿到响应源码，使用 response.text （但是注意只有TextResponse及其子类才有text数据成员）。（5） flags ，list，一个包含 Response.flags 属性的列表。传递给response对象时使用的是浅拷贝。（6） request ，scrapy.Request类或其子类的实例，就是用于生成response的request对象。（7） certificate ，twisted.internet.ssl.Certificate，SSL证书。（8） ip_address ，ipaddress.IPv4Address或者ipaddress.IPv6Address，表示响应来源网址的ip地址。（9） protocol ，str，下载响应的协议，比如 "HTTP/1.0" ， "HTTP/1.1" 或者 "h2" 。response对象有以下成员：（1） url ，只读，可用 replace() 方法更改。（2） status 。（3） headers ，可用 response.headers.getlist('键名') 或 get() 方法来取出响应头中的某项值。（4） body ，只读，可用 replace() 方法更改。（5） request ，发生http重定向时，原来的request会和重定向后的response关联；这个属性只应该在爬虫文件、爬虫中间件中使用，不要在下载中间件里调用。（6） meta ，从request对象传递过来的超数据，即使发生重定向或网页retry也不会改变关联。（7） cb_kwargs ，scrapy 2.0版本后新增，从request对象那里拿到的字典参数表，会传递给回调函数。它也不受重定向和retry影响。（8） flags ，表示response状态，比如 "cached" 、 "redirected" ，会用于logging。（9） certificate 。（10） ip_address 。（11） protocol 。（12） attributes: Tuple[str, ...]= ('url', 'status', 'headers', 'body', 'flags', 'request', 'certificate', 'ip_address', 'protocol') ，一个字符串元组，包含response对象的信息，被 replace() 方法调用。（13） copy() ，返回当前response对象的浅拷贝。（14） replace([url, status, headers, body, request, flags, cls]) ，把提到的值更改，其余值不变，返回新的response对象，同样是浅拷贝。（15） urljoin(url) ，对于互相关联的url，使用这个方法会把相对url改写成绝对url。它实际上是使用了 urllib.parse.urljoin(response.url, url) 这个方法。（16） follow(url, callback=None, method='GET', headers=None, body=None, cookies=None, meta=None, encoding='utf-8', priority=0, dont_filter=False, errback=None, cb_kwargs=None, flags=None)→ Request ，返回一个request对象，参数和request对象的构造方法一样，不同在于url参数可以使用相对url，或者scrapy.link.Link对象。（17） follow_all(urls, callback=None, method='GET', headers=None, body=None, cookies=None, meta=None, encoding='utf-8', priority=0, dont_filter=False, errback=None, cb_kwargs=None, flags=None)→ Generator[Request, None, None] ，类似 follow() 方法。下面介绍一些response的子类：（1）scrapy.http.TextResponse，添加了编码方式，可用于图像、声音等媒体文件爬取。最主要的区别是在构造函数中有一个参数 encoding ，str，指定响应的解析格式，为None时会自动匹配。另外它还多出了一些属性、方法，包括 text 、 encoding 、 selector 、 xpath() 、 css() 、 json() （将json文件转换为python对象）。（2）scrapy.http.HtmlResponse和scrapy.http.XmlResponse，它们都是TextResponse的子类。12.link extractorlink extractor对象可以从response对象中提取链接，可用于全站爬取。这在前面介绍CrawlSpider类时提到过。除了用于CrawlSpider爬虫类，也可以用于其他爬虫，比如用 LxmlLinkExtractor.extract_links 方法得到响应对象里所有符合条件的link对象列表。它的构造函数为 scrapy.linkextractors.lxmlhtml.LxmlLinkExtractor(allow=(), deny=(), allow_domains=(), deny_domains=(), deny_extensions=None, restrict_xpaths=(), restrict_css=(), tags=('a', 'area'), attrs=('href',), canonicalize=False, unique=True, process_value=None, strip=True) ，参数有：（1） allow ，str或list，一个re字符串或re字符串列表，提取的链接必须满足这些re规则。不给出或为空字符串时，表示对链接不做re限制。（2） deny ，str或list，用于给出排除链接的re规则，它的优先级高于 allow （先排除后提取）。也可以不给出或为空str。（3） allow_domains ，str或list，str或str列表，只有response.url在这里面时才会用于提取链接。（4） deny_domains ，和 allow_domains 相反。（5） deny_extensions ，str列表，用于给出提取链接时忽略的extension，默认由 scrapy.linkextractors.IGNORED_EXTENSIONS 指定。在scrapy 2.0版本后， IGNORED_EXTENSIONS 包括 7z, 7zip, apk, bz2, cdr, dmg, ico, iso, tar, tar.gz, webm, 和 xz。（6） restrict_xpaths ，str或list，只在这些xpath表达式选中的节点里提取链接。（7） restrict_css ，str或list，功能同上。（8） restrict_text ，str或list，和 allow 类似，不同之处在于各个re规则之间是 或 的关系，也就是链接只要满足其中一条re规则就可以被提取。（9） tags ，str或list，用于提取链接的标签名必须在这里给出的标签名中。常用的标签名有 a ， area 。（10） attrs ，list，用于提取链接的属性，默认是 'href' 。（11） canonicalize ，bool，是否对被提取的链接使用 w3lib.url.canonicalize_url 来规范化，默认为False，不推荐改动这个值。（12） unique ，bool，是否在提取链接这个环节就去重。（13） process_value ，collections.abc.Callable，它是一个函数，用于对提取的链接进行处理，默认是 lambda x: x 。使用例：要提取链接 Link text ，使用：def process_value(value): m = re.search("javascript:goToPage\('(.*?)'", value) if m: return m.group(1)（14） strip ，bool，是否对提取到的链接str运行 strip() 方法，这个不需要改动，因为有些标准要求url前后不得有空白字符。LxmlLinkExtractor还有 extract_links(response) 方法，用于将提取规则应用于response对象、返回符合规则的link对象列表。link对象，构造函数为 scrapy.link.Link(url, text='', fragment='', nofollow=False) ，表示用linkextractor提取出的链接对象。原文给出的参数例子：Dont follow this one Parameters url – the absolute url being linked to in the anchor tag. From the sample, this is https://example.com/nofollow.html. text – the text in the anchor tag. From the sample, this is Dont follow this one. fragment – the part of the url after the hash symbol. From the sample, this is foo. nofollow – an indication of the presence or absence of a nofollow value in the rel attribute of the anchor tag.13.settings这部分内容参考 Python，爬虫与深度学习（11）——scrapy中settings的可选项 或原文。下面是一些没有提到的可选项：AJAXCRAWL_ENABLED AUTOTHROTTLE_DEBUG AUTOTHROTTLE_ENABLED AUTOTHROTTLE_MAX_DELAY AUTOTHROTTLE_START_DELAY AUTOTHROTTLE_TARGET_CONCURRENCY CLOSESPIDER_ERRORCOUNT CLOSESPIDER_ITEMCOUNT CLOSESPIDER_PAGECOUNT CLOSESPIDER_TIMEOUT COMMANDS_MODULE COMPRESSION_ENABLED COOKIES_DEBUG COOKIES_ENABLED FEEDS FEED_EXPORTERS FEED_EXPORTERS_BASE FEED_EXPORT_BATCH_ITEM_COUNT FEED_EXPORT_ENCODING FEED_EXPORT_FIELDS FEED_EXPORT_INDENT FEED_STORAGES FEED_STORAGES_BASE FEED_STORAGE_FTP_ACTIVE FEED_STORAGE_S3_ACL FEED_STORE_EMPTY FEED_URI_PARAMS FILES_EXPIRES FILES_RESULT_FIELD FILES_STORE FILES_STORE_GCS_ACL FILES_STORE_S3_ACL FILES_URLS_FIELD HTTPCACHE_ALWAYS_STORE HTTPCACHE_DBM_MODULE HTTPCACHE_DIR HTTPCACHE_ENABLED HTTPCACHE_EXPIRATION_SECS HTTPCACHE_GZIP HTTPCACHE_IGNORE_HTTP_CODES HTTPCACHE_IGNORE_MISSING HTTPCACHE_IGNORE_RESPONSE_CACHE_CONTROLS HTTPCACHE_IGNORE_SCHEMES HTTPCACHE_POLICY HTTPCACHE_STORAGE HTTPERROR_ALLOWED_CODES HTTPERROR_ALLOW_ALL HTTPPROXY_AUTH_ENCODING HTTPPROXY_ENABLED IMAGES_EXPIRES IMAGES_MIN_HEIGHT IMAGES_MIN_WIDTH IMAGES_RESULT_FIELD IMAGES_STORE IMAGES_STORE_GCS_ACL IMAGES_STORE_S3_ACL IMAGES_THUMBS IMAGES_URLS_FIELD MAIL_FROM MAIL_HOST MAIL_PASS MAIL_PORT MAIL_SSL MAIL_TLS MAIL_USER MEDIA_ALLOW_REDIRECTS METAREFRESH_ENABLED METAREFRESH_IGNORE_TAGS METAREFRESH_MAXDELAY REDIRECT_ENABLED REDIRECT_MAX_TIMES REFERER_ENABLED REFERRER_POLICY RETRY_ENABLED RETRY_HTTP_CODES RETRY_PRIORITY_ADJUST RETRY_TIMES TELNETCONSOLE_HOST TELNETCONSOLE_PASSWORD TELNETCONSOLE_PORT TELNETCONSOLE_USERNAME14.Exceptionsscrapy提供了一些exception类。有些不太能翻译，就照搬原文了。（1） scrapy.exceptions.CloseSpider(reason='cancelled') ，爬虫的回调函数要求爬虫关闭、停止时会引发这个错误。 reason 参数用于描述关闭原因。使用例：def parse_page(self, response): if 'Bandwidth exceeded' in response.body: raise CloseSpider('bandwidth_exceeded')（2） scrapy.exceptions.DontCloseSpider，This exception can be raised in a spider_idle signal handler to prevent the spider from being closed.（3） scrapy.exceptions.DropItem ，这个错误是在管道处理数据时，如果item不符合要求，可以手动raise这个错误来放弃item。（4） scrapy.exceptions.IgnoreRequest ，This exception can be raised by the Scheduler or any downloader middleware to indicate that the request should be ignored.（5） scrapy.exceptions.NotConfigured ，This exception can be raised by some components to indicate that they will remain disabled. Those components include:Extensions Item pipelines Downloader middlewares Spider middlewaresThe exception must be raised in the component’s __init__ method.（6） scrapy.exceptions.NotSupported ，This exception is raised to indicate an unsupported feature.（7） scrapy.exceptions.StopDownload(fail=True) ，Raised from a bytes_received or headers_received signal handler to indicate that no further bytes should be downloaded for a response.The fail boolean parameter controls which method will handle the resulting response:If fail=True (default), the request errback is called. The response object is available as the response attribute of the StopDownload exception, which is in turn stored as the value attribute of the received Failure object. This means that in an errback defined as def errback(self, failure), the response can be accessed though failure.value.response.If fail=False, the request callback is called instead.In both cases, the response could have its body truncated: the body contains all bytes received up until the exception is raised, including the bytes received in the signal handler that raises the exception. Also, the response object is marked with "download_stopped" in its Response.flags attribute.Note: fail is a keyword-only parameter, i.e. raising StopDownload(False) or StopDownload(True) will raise a TypeError.See the documentation for the bytes_received and headers_received signals and the Stopping the download of a Response topic for additional information and examples.15.loggingscrapy.log是从python自带的logging衍生而来，但是舍弃了很多部分。logging模块里有5种log等级，从高到低依次为： logging.CRITICAL ， logging.ERROR ， logging.WARNING ， logging.INFO ， logging.DEBUG 。原生的logging模块使用例如下：import logging logging.warning("This is a warning") logging.log(logging.WARNING, "This is a warning")使用logging模块时，常用的做法是为每个模块单独开一个log文件（单独的log handler），如下：import logging logger = logging.getLogger(__name__) logger.warning("This is a warning")在scrapy中，爬虫对象有自己的logger，使用例如下：import scrapy class MySpider(scrapy.Spider): name = 'myspider' start_urls = ['https://scrapy.org'] def parse(self, response): self.logger.info('Parse function called on %s', response.url)这个logger handler使用的是爬虫的name。然后，在settings.py文件中对log进行一些设置，可选项有：LOG_FILE LOG_FILE_APPEND LOG_ENABLED LOG_ENCODING LOG_LEVEL LOG_FORMAT LOG_DATEFORMAT LOG_STDOUT LOG_SHORT_NAMES详细的解释见 Python，爬虫与深度学习（11）——scrapy中settings的可选项 ，其余的就不多介绍了。16.Stats Collection收集爬虫的状态信息，这个部分在分布式爬虫中用的更多，但是有现成的分布式scrapy（scrapy-redis、scrapyd），这里就不介绍了。17.Sending e-mail简单来说就是scrapy提供了发送邮件的功能。但是其实python自己的smtplib用来发邮件绰绰有余，另外邮件更多用于分布式，最后即使是分布式scrapy更多用的也是网页UI，方便快捷，笔者感觉发送邮件的功能有些鸡肋（除非你需要在上班时摸鱼查看爬虫发送来的邮件），但还是介绍一下。使用例如下：from scrapy.mail import MailSender mailer = MailSender() #使用构造函数生成 mailer = MailSender.from_settings(settings) #使用from_settings方法，从settings生成 mailer.send(to=["someone@example.com"], subject="Some subject", body="Some body", cc=["another@example.com"]) #发送邮件的方法MailSender的构造函数为 scrapy.mail.MailSender(smtphost=None, mailfrom=None, smtpuser=None, smtppass=None, smtpport=None) ，参数有：（1） smtphost (str or bytes) – the SMTP host to use for sending the emails. If omitted, the MAIL_HOST setting will be used.（2） mailfrom (str) – the address used to send emails (in the From: header). If omitted, the MAIL_FROM setting will be used.（3） smtpuser – the SMTP user. If omitted, the MAIL_USER setting will be used. If not given, no SMTP authentication will be performed.（4） smtppass (str or bytes) – the SMTP pass for authentication.（5） smtpport (int) – the SMTP port to connect to（6） smtptls (bool) – enforce using SMTP STARTTLS（7） smtpssl (bool) – enforce using a secure SSL connection如果对smtp邮件服务感到陌生，可参考 Docker树莓派实践——Typecho及其备份 这篇文章中typecho的CommentNotifier插件这部分入门。18.Telnet Consolescrapy提供了telent控制台来连接、操作scrapy项目，这在分布式中用的更多，这里也不介绍了。19.FAQ官方文档提供的一些Frequently Asked Questions，复制如下：How does Scrapy compare to BeautifulSoup or lxml?¶ BeautifulSoup and lxml are libraries for parsing HTML and XML. Scrapy is an application framework for writing web spiders that crawl web sites and extract data from them. Scrapy provides a built-in mechanism for extracting data (called selectors) but you can easily use BeautifulSoup (or lxml) instead, if you feel more comfortable working with them. After all, they’re just parsing libraries which can be imported and used from any Python code. In other words, comparing BeautifulSoup (or lxml) to Scrapy is like comparing jinja2 to Django. Can I use Scrapy with BeautifulSoup?¶ Yes, you can. As mentioned above, BeautifulSoup can be used for parsing HTML responses in Scrapy callbacks. You just have to feed the response’s body into a BeautifulSoup object and extract whatever data you need from it. Here’s an example spider using BeautifulSoup API, with lxml as the HTML parser: from bs4 import BeautifulSoup import scrapy class ExampleSpider(scrapy.Spider): name = "example" allowed_domains = ["example.com"] start_urls = ( 'http://www.example.com/', ) def parse(self, response): # use lxml to get decent HTML parsing speed soup = BeautifulSoup(response.text, 'lxml') yield { "url": response.url, "title": soup.h1.string } Note BeautifulSoup supports several HTML/XML parsers. See BeautifulSoup’s official documentation on which ones are available. Did Scrapy “steal” X from Django?¶ Probably, but we don’t like that word. We think Django is a great open source project and an example to follow, so we’ve used it as an inspiration for Scrapy. We believe that, if something is already done well, there’s no need to reinvent it. This concept, besides being one of the foundations for open source and free software, not only applies to software but also to documentation, procedures, policies, etc. So, instead of going through each problem ourselves, we choose to copy ideas from those projects that have already solved them properly, and focus on the real problems we need to solve. We’d be proud if Scrapy serves as an inspiration for other projects. Feel free to steal from us! Does Scrapy work with HTTP proxies?¶ Yes. Support for HTTP proxies is provided (since Scrapy 0.8) through the HTTP Proxy downloader middleware. See HttpProxyMiddleware. How can I scrape an item with attributes in different pages?¶ See Passing additional data to callback functions. Scrapy crashes with: ImportError: No module named win32api¶ You need to install pywin32 because of this Twisted bug. How can I simulate a user login in my spider?¶ See Using FormRequest.from_response() to simulate a user login. Does Scrapy crawl in breadth-first or depth-first order?¶ By default, Scrapy uses a LIFO queue for storing pending requests, which basically means that it crawls in DFO order. This order is more convenient in most cases. If you do want to crawl in true BFO order, you can do it by setting the following settings: DEPTH_PRIORITY = 1 SCHEDULER_DISK_QUEUE = 'scrapy.squeues.PickleFifoDiskQueue' SCHEDULER_MEMORY_QUEUE = 'scrapy.squeues.FifoMemoryQueue' While pending requests are below the configured values of CONCURRENT_REQUESTS, CONCURRENT_REQUESTS_PER_DOMAIN or CONCURRENT_REQUESTS_PER_IP, those requests are sent concurrently. As a result, the first few requests of a crawl rarely follow the desired order. Lowering those settings to 1 enforces the desired order, but it significantly slows down the crawl as a whole. My Scrapy crawler has memory leaks. What can I do?¶ See Debugging memory leaks. Also, Python has a builtin memory leak issue which is described in Leaks without leaks. How can I make Scrapy consume less memory?¶ See previous question. How can I prevent memory errors due to many allowed domains?¶ If you have a spider with a long list of allowed_domains (e.g. 50,000+), consider replacing the default OffsiteMiddleware spider middleware with a custom spider middleware that requires less memory. For example: If your domain names are similar enough, use your own regular expression instead joining the strings in allowed_domains into a complex regular expression. If you can meet the installation requirements, use pyre2 instead of Python’s re to compile your URL-filtering regular expression. See issue 1908. See also other suggestions at StackOverflow. Note Remember to disable scrapy.spidermiddlewares.offsite.OffsiteMiddleware when you enable your custom implementation: SPIDER_MIDDLEWARES = { 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware': None, 'myproject.middlewares.CustomOffsiteMiddleware': 500, } Can I use Basic HTTP Authentication in my spiders?¶ Yes, see HttpAuthMiddleware. Why does Scrapy download pages in English instead of my native language?¶ Try changing the default Accept-Language request header by overriding the DEFAULT_REQUEST_HEADERS setting. Where can I find some example Scrapy projects?¶ See Examples. Can I run a spider without creating a project?¶ Yes. You can use the runspider command. For example, if you have a spider written in a my_spider.py file you can run it with: scrapy runspider my_spider.py See runspider command for more info. I get “Filtered offsite request” messages. How can I fix them?¶ Those messages (logged with DEBUG level) don’t necessarily mean there is a problem, so you may not need to fix them. Those messages are thrown by the Offsite Spider Middleware, which is a spider middleware (enabled by default) whose purpose is to filter out requests to domains outside the ones covered by the spider. For more info see: OffsiteMiddleware. What is the recommended way to deploy a Scrapy crawler in production?¶ See Deploying Spiders. Can I use JSON for large exports?¶ It’ll depend on how large your output is. See this warning in JsonItemExporter documentation. Can I return (Twisted) deferreds from signal handlers?¶ Some signals support returning deferreds from their handlers, others don’t. See the Built-in signals reference to know which ones. What does the response status code 999 means?¶ 999 is a custom response status code used by Yahoo sites to throttle requests. Try slowing down the crawling speed by using a download delay of 2 (or higher) in your spider: class MySpider(CrawlSpider): name = 'myspider' download_delay = 2 # [ ... rest of the spider code ... ] Or by setting a global download delay in your project with the DOWNLOAD_DELAY setting. Can I call pdb.set_trace() from my spiders to debug them?¶ Yes, but you can also use the Scrapy shell which allows you to quickly analyze (and even modify) the response being processed by your spider, which is, quite often, more useful than plain old pdb.set_trace(). For more info see Invoking the shell from spiders to inspect responses. Simplest way to dump all my scraped items into a JSON/CSV/XML file?¶ To dump into a JSON file: scrapy crawl myspider -O items.json To dump into a CSV file: scrapy crawl myspider -O items.csv To dump into a XML file: scrapy crawl myspider -O items.xml For more information see Feed exports What’s this huge cryptic __VIEWSTATE parameter used in some forms?¶ The __VIEWSTATE parameter is used in sites built with ASP.NET/VB.NET. For more info on how it works see this page. Also, here’s an example spider which scrapes one of these sites. What’s the best way to parse big XML/CSV data feeds?¶ Parsing big feeds with XPath selectors can be problematic since they need to build the DOM of the entire feed in memory, and this can be quite slow and consume a lot of memory. In order to avoid parsing all the entire feed at once in memory, you can use the functions xmliter and csviter from scrapy.utils.iterators module. In fact, this is what the feed spiders (see Spiders) use under the cover. Does Scrapy manage cookies automatically?¶ Yes, Scrapy receives and keeps track of cookies sent by servers, and sends them back on subsequent requests, like any regular web browser does. For more info see Requests and Responses and CookiesMiddleware. How can I see the cookies being sent and received from Scrapy?¶ Enable the COOKIES_DEBUG setting. How can I instruct a spider to stop itself?¶ Raise the CloseSpider exception from a callback. For more info see: CloseSpider. How can I prevent my Scrapy bot from getting banned?¶ See Avoiding getting banned. Should I use spider arguments or settings to configure my spider?¶ Both spider arguments and settings can be used to configure your spider. There is no strict rule that mandates to use one or the other, but settings are more suited for parameters that, once set, don’t change much, while spider arguments are meant to change more often, even on each spider run and sometimes are required for the spider to run at all (for example, to set the start url of a spider). To illustrate with an example, assuming you have a spider that needs to log into a site to scrape data, and you only want to scrape data from a certain section of the site (which varies each time). In that case, the credentials to log in would be settings, while the url of the section to scrape would be a spider argument. I’m scraping a XML document and my XPath selector doesn’t return any items¶ You may need to remove namespaces. See Removing namespaces. How to split an item into multiple items in an item pipeline?¶ Item pipelines cannot yield multiple items per input item. Create a spider middleware instead, and use its process_spider_output() method for this purpose. For example: from copy import deepcopy from itemadapter import is_item, ItemAdapter class MultiplyItemsMiddleware: def process_spider_output(self, response, result, spider): for item in result: if is_item(item): adapter = ItemAdapter(item) for _ in range(adapter['multiply_by']): yield deepcopy(item) Does Scrapy support IPv6 addresses?¶ Yes, by setting DNS_RESOLVER to scrapy.resolver.CachingHostnameResolver. Note that by doing so, you lose the ability to set a specific timeout for DNS requests (the value of the DNS_TIMEOUT setting is ignored). How to deal with <class 'ValueError'>: filedescriptor out of range in select() exceptions?¶ This issue has been reported to appear when running broad crawls in macOS, where the default Twisted reactor is twisted.internet.selectreactor.SelectReactor. Switching to a different reactor is possible by using the TWISTED_REACTOR setting. How can I cancel the download of a given response?¶ In some situations, it might be useful to stop the download of a certain response. For instance, sometimes you can determine whether or not you need the full contents of a response by inspecting its headers or the first bytes of its body. In that case, you could save resources by attaching a handler to the bytes_received or headers_received signals and raising a StopDownload exception. Please refer to the Stopping the download of a Response topic for additional information and examples. Running runspider I get error: No spider found in file: <filename>¶ This may happen if your Scrapy project has a spider module with a name that conflicts with the name of one of the Python standard library modules, such as csv.py or os.py, or any Python package that you have installed. See issue 2680.
步骤/目录： 1.介绍 （1）sitemap （2）robots.txt 2.使用scrapy生成sitemap本文首发于个人博客https://lisper517.top/index.php/archives/50/，转载请注明出处。本文的目的是介绍sitemap与robots.txt。本文写作日期为2022年9月14日。运行的平台为win10，编辑器为VS Code。爬虫写的好，牢房进的早。使用爬虫时不要占用服务器太多流量，否则可能惹祸上身。笔者的能力有限，也并非CS从业人员，很多地方难免有纰漏或者不符合代码原则的地方，请在评论中指出。1.介绍如果感觉这些概念有些抽象，可以访问一些网址看看范例，比如 https://www.baidu.com/robots.txt 、 https://img.kuaidaili.com/sitemap.xml 等。（1）sitemapsitemap是一个站点的网页地图，它可以帮助爬虫，给出哪些网页可供抓取（sitemap应该只包含允许爬取的网页）、和各网址的基础数据（更新时间、更新频率、重要程度等）。因为搜索引擎都需要爬虫来获取网页的详情，用sitemap可以做SEO（search engine optimization，搜索引擎优化），这样baidu、google等各大搜索引擎的爬虫能爬到自己网页的页面，则在搜索关键字时自己网页出现的频率也会升高。sitemap的格式等信息可以到 sitemap官网 查看，下面是一份范例：<?xml version="1.0" encoding="UTF-8"?> <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"> <url> <loc>http://www.example.com/</loc> <lastmod>2005-01-01</lastmod> <changefreq>monthly</changefreq> <priority>0.8</priority> </url> <url> <loc>http://www.example.com/.../.../</loc> <lastmod>2005-01-01</lastmod> <changefreq>monthly</changefreq> <priority>0.8</priority> </url> ... </urlset> 一个编写好的sitemap，一般会放在网站的根目录下，命名为sitemap.xml。另外，除了被动等待自己的网页被搜索引擎爬取，还可以自己向搜索引擎提交sitemap；wordpress等模板在建站后会自动生成sitemap。（2）robots.txtrobots.txt也是一个建议爬虫如何爬取自己网站的文件，同样也放在网站根目录下。它的格式为：User-agent: * Allow: / Disallow: / Sitemap: http://www.***.com/sitemap.xmlUser-agent后面用于指定搜索引擎爬虫，比如Googlebot、Baiduspider、， * 则表示对所有爬虫适用；Allow、Disallow后面跟允许或禁止爬取的目录，只写目录名不用写全网址；Sitemap则是指明自己网站的sitemap在哪里，这个只用写一次（User-agent、Allow和Disallow则可能出现多次）。另外，如果写成：User-agent: Googlebot Allow: / User-agent: * Disallow: /就是允许google爬虫访问自己的全部页面，但禁止其他爬虫访问自己的全部页面。这个例子的重点在于前面指明单个爬虫，后面用通配符*表示其他所有爬虫。2.使用scrapy生成sitemap首先要提醒，这种方式其实和爬虫软件去爬取你的页面一样。如果你的网站，网页之间内链不多，建议还是从网站的根目录或者数据库里生成sitemap。安装好python、scrapy环境后，在win10机器上的cmd中使用如下命令：cd 放scrapy爬虫的目录 scrapy startproject sitemap cd sitemap scrapy genspider sitemap_mysite example.com然后开始编辑scrapy项目。items.py如下：# Define here the models for your scraped items # # See documentation in: # https://docs.scrapy.org/en/latest/topics/items.html import scrapy class SitemapItem(scrapy.Item): # define the fields for your item here like: # name = scrapy.Field() url = scrapy.Field() middlewares.py如下：# Define here the models for your spider middleware # # See documentation in: # https://docs.scrapy.org/en/latest/topics/spider-middleware.html import random from scrapy import signals # useful for handling different item types with a single interface from itemadapter import is_item, ItemAdapter class SitemapSpiderMiddleware: # Not all methods need to be defined. If a method is not defined, # scrapy acts as if the spider middleware does not modify the # passed objects. @classmethod def from_crawler(cls, crawler): # This method is used by Scrapy to create your spiders. s = cls() crawler.signals.connect(s.spider_opened, signal=signals.spider_opened) return s def process_spider_input(self, response, spider): # Called for each response that goes through the spider # middleware and into the spider. # Should return None or raise an exception. return None def process_spider_output(self, response, result, spider): # Called with the results returned from the Spider, after # it has processed the response. # Must return an iterable of Request, or item objects. for i in result: yield i def process_spider_exception(self, response, exception, spider): # Called when a spider or process_spider_input() method # (from other spider middleware) raises an exception. # Should return either None or an iterable of Request or item objects. pass def process_start_requests(self, start_requests, spider): # Called with the start requests of the spider, and works # similarly to the process_spider_output() method, except # that it doesn’t have a response associated. # Must return only requests (not items). for r in start_requests: yield r def spider_opened(self, spider): spider.logger.info('Spider opened: %s' % spider.name) class SitemapDownloaderMiddleware: # Not all methods need to be defined. If a method is not defined, # scrapy acts as if the downloader middleware does not modify the # passed objects. UA_list = [ 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36', 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36 Edg/105.0.1343.33', 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:104.0) Gecko/20100101 Firefox/104.0' ] @classmethod def from_crawler(cls, crawler): # This method is used by Scrapy to create your spiders. s = cls() crawler.signals.connect(s.spider_opened, signal=signals.spider_opened) return s def process_request(self, request, spider): # Called for each request that goes through the downloader # middleware. # Must either: # - return None: continue processing this request # - or return a Response object # - or return a Request object # - or raise IgnoreRequest: process_exception() methods of # installed downloader middleware will be called request.headers['User-Agent'] = random.choice(self.UA_list) return None def process_response(self, request, response, spider): # Called with the response returned from the downloader. # Must either; # - return a Response object # - return a Request object # - or raise IgnoreRequest return response def process_exception(self, request, exception, spider): # Called when a download handler or a process_request() # (from other downloader middleware) raises an exception. # Must either: # - return None: continue processing this exception # - return a Response object: stops process_exception() chain # - return a Request object: stops process_exception() chain pass def spider_opened(self, spider): spider.logger.info('Spider opened: %s' % spider.name) pipelines.py如下：# Define your item pipelines here # # Don't forget to add your pipeline to the ITEM_PIPELINES setting # See: https://docs.scrapy.org/en/latest/topics/item-pipeline.html # useful for handling different item types with a single interface from itemadapter import ItemAdapter import os class SitemapPipeline: def open_spider(self, spider): file_name = 'urls.txt' abs_file_path = os.path.join(os.getcwd(), file_name) self.fp = open(abs_file_path, 'a', encoding='utf-8') def process_item(self, item, spider): url = item['url'] self.fp.write(url + '\n') return item def close_spider(self, spider): self.fp.close()settings.py如下：# Scrapy settings for sitemap project # # For simplicity, this file contains only settings considered important or # commonly used. You can find more settings consulting the documentation: # # https://docs.scrapy.org/en/latest/topics/settings.html # https://docs.scrapy.org/en/latest/topics/downloader-middleware.html # https://docs.scrapy.org/en/latest/topics/spider-middleware.html LOG_LEVEL = 'ERROR' BOT_NAME = 'sitemap' SPIDER_MODULES = ['sitemap.spiders'] NEWSPIDER_MODULE = 'sitemap.spiders' # Crawl responsibly by identifying yourself (and your website) on the user-agent #USER_AGENT = 'sitemap (+http://www.yourdomain.com)' # Obey robots.txt rules ROBOTSTXT_OBEY = False # Configure maximum concurrent requests performed by Scrapy (default: 16) #CONCURRENT_REQUESTS = 32 # Configure a delay for requests for the same website (default: 0) # See https://docs.scrapy.org/en/latest/topics/settings.html#download-delay # See also autothrottle settings and docs #DOWNLOAD_DELAY = 3 # The download delay setting will honor only one of: #CONCURRENT_REQUESTS_PER_DOMAIN = 16 #CONCURRENT_REQUESTS_PER_IP = 16 # Disable cookies (enabled by default) #COOKIES_ENABLED = False # Disable Telnet Console (enabled by default) #TELNETCONSOLE_ENABLED = False # Override the default request headers: #DEFAULT_REQUEST_HEADERS = { # 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', # 'Accept-Language': 'en', #} # Enable or disable spider middlewares # See https://docs.scrapy.org/en/latest/topics/spider-middleware.html #SPIDER_MIDDLEWARES = { # 'sitemap.middlewares.SitemapSpiderMiddleware': 543, #} # Enable or disable downloader middlewares # See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html DOWNLOADER_MIDDLEWARES = { 'sitemap.middlewares.SitemapDownloaderMiddleware': 543, } # Enable or disable extensions # See https://docs.scrapy.org/en/latest/topics/extensions.html #EXTENSIONS = { # 'scrapy.extensions.telnet.TelnetConsole': None, #} # Configure item pipelines # See https://docs.scrapy.org/en/latest/topics/item-pipeline.html ITEM_PIPELINES = { 'sitemap.pipelines.SitemapPipeline': 300, } # Enable and configure the AutoThrottle extension (disabled by default) # See https://docs.scrapy.org/en/latest/topics/autothrottle.html #AUTOTHROTTLE_ENABLED = True # The initial download delay #AUTOTHROTTLE_START_DELAY = 5 # The maximum download delay to be set in case of high latencies #AUTOTHROTTLE_MAX_DELAY = 60 # The average number of requests Scrapy should be sending in parallel to # each remote server #AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0 # Enable showing throttling stats for every response received: #AUTOTHROTTLE_DEBUG = False # Enable and configure HTTP caching (disabled by default) # See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html#httpcache-middleware-settings #HTTPCACHE_ENABLED = True #HTTPCACHE_EXPIRATION_SECS = 0 #HTTPCACHE_DIR = 'httpcache' #HTTPCACHE_IGNORE_HTTP_CODES = [] #HTTPCACHE_STORAGE = 'scrapy.extensions.httpcache.FilesystemCacheStorage' 最后是sitemap_mysite.py：import scrapy, re from sitemap.items import SitemapItem class SitemapMysiteSpider(scrapy.Spider): name = 'sitemap_mysite' #allowed_domains = ['example.com'] start_urls = ['https://lisper517.top/'] def parse(self, response): print(response.url) item = SitemapItem() item['url'] = response.url yield item other_urls = re.findall(r'"https://lisper517.top/.+?"', response.text) other_urls = [other_urls[i][1:-1] for i in range(len(other_urls))] for other_url in other_urls: if not other_url.startswith('https://lisper517.top/resources'): yield scrapy.Request(url=other_url, callback=self.parse)可自行把笔者的网址更改为自己的站点。在cmd中，使用 scrapy crawl sitemap_mysite 运行这个爬虫，在当前目录下会生成 urls.txt 文件，可以在这里查看自己的网站上有哪些站点。根据这份文件，生成自己的sitemap.xml和robots.txt，放到网站的根目录下即可。
步骤/目录： 1.settings的指定方法 2.setting中的常用设置与默认值 （1）爬虫效率相关 （2）字典类设置 （3）log相关 （4）其他的重要设置 （5）其他的不重要设置本文首发于个人博客https://lisper517.top/index.php/archives/49/，转载请注明出处。本文的目的是介绍scrapy的settings的可选项及使用方法。本文写作日期为2022年9月14日，主要参考了 scrapy官方文档 。爬虫写的好，牢房进的早。使用爬虫时不要占用服务器太多流量，否则可能惹祸上身。笔者的能力有限，也并非CS从业人员，很多地方难免有纰漏或者不符合代码原则的地方，请在评论中指出。1.settings的指定方法一共有5种指定settings的方法：（1）爬虫单独指定，就是在爬虫类下面用 custom_settings 数据成员指定，如：class MySpider(scrapy.Spider): name = 'myspider' custom_settings = { 'CONCURRENT_REQUESTS': 1, 'DOWNLOAD_DELAY': 10, 'CONCURRENT_REQUESTS_PER_DOMAIN': 1 }这种方法适合有多个爬虫、多个网站，但有些网站需要调整设置的时候，比较常用。（2）命令行指定，即 scrapy crawl myspider -s LOG_FILE=scrapy.log 这样的格式，缺点是能指定的项比较少，只能用来覆盖部分settings.py里的设置。另外的三种方法比较进阶，这里就不介绍了。最后，在爬虫类里，可以通过 self.settings 调出爬虫的设置。2.setting中的常用设置与默认值这里提到的settings可选项只是很少一部分，其他的要不就是更加进阶，要不就是一般不会用到。（1）爬虫效率相关CONCURRENT_ITEMS=100 ，设置每个响应拿到后有多少个item pipeline同时运行。由于爬虫的瓶颈一般在网络，这个设置没有太大用处。CONCURRENT_REQUESTS=16 ，下载器最多能同时发送多少个请求。对于一些封ip的网站，需要用这个设置。CONCURRENT_REQUESTS_PER_DOMAIN=8 ，对一个域名最多发送多少请求。对于一些封ip的网站，可以用这个设置。CONCURRENT_REQUESTS_PER_IP=0 ，对一个ip最多发送多少请求。如果设置了这个，则 CONCURRENT_REQUESTS_PER_DOMAIN 会失效。DOWNLOAD_DELAY=0 ，用于设置从同一网站爬取连续页面时的等待时间，单位为秒（可以用浮点数）。对于一些封ip的网站，需要用这个设置。这个等待时间不是定死的，实际的时间会在0.5倍~1.5倍 DOWNLOAD_DELAY 之间浮动（若不想浮动，用 RANDOMIZE_DOWNLOAD_DELAY=False 来指定，但一般不要这样做，因为有些网站通过分析请求间隔来判别爬虫）。除了在爬虫类中用 custom_settings 指定、在 settings.py 里指定，还可以通过爬虫的 download_delay 属性来指定。总结一下，如果有些网站会封访问频繁的ip，可以在爬虫类里使用： custom_settings = { 'CONCURRENT_REQUESTS': 1, 'DOWNLOAD_DELAY': 10, 'CONCURRENT_REQUESTS_PER_DOMAIN': 1 }这样的设置来限制访问频率。（2）字典类设置这些设置一般都是指定中间件等类，和它们的优先级（或者顺序），一般越小优先级越高。DOWNLOADER_MIDDLEWARES ，用于设置下载中间件及其优先级。EXTENSIONS ，类似 DOWNLOADER_MIDDLEWARES ，用于指定扩展及其优先级。ITEM_PIPELINES ，用于指定item pipelines及其优先级。（3）log相关scrapy使用的是python自带的logging模块，所以有些设置需要参考logging。LOG_ENABLED=True ，是否允许记录log。LOG_ENCODING='utf-8' ，log的字符集。LOG_FILE=None ，logging输出的文件名。为None时，使用standard error。LOG_FILE_APPEND=True ，如果为False，每次运行时会覆盖 LOG_FILE 指定的log文件。LOG_FORMAT='%(asctime)s [%(name)s] %(levelname)s: %(message)s' ，指定每条log的格式。想自己设置格式请参考logging模块的格式说明。LOG_DATEFORMAT='%Y-%m-%d %H:%M:%S' ，用于指定 LOG_FORMAT 设置中 %(asctime)s 的输出格式。LOG_LEVEL='DEBUG' ，设置log显示的最低级别。从高到低为CRITICAL, ERROR, WARNING, INFO, DEBUG。LOG_STDOUT=False ，如果为True，会将所有输出重定向到log，比如python的print函数。（4）其他的重要设置DEFAULT_REQUEST_HEADERS 设置请求头，默认为：{ 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', 'Accept-Language': 'en', }这里可以设置一些固定的项。如果想设置user-agent，一般在middlrewares.py的下载中间件里设置；这里不能设置cookies，而应该在下载中间件的子类CookiesMiddleware中对 request.cookies 赋值来进行设置，例子可参考 这篇文章 。ROBOTSTXT_OBEY=True ，如果为False，则scrapy不会遵守网站的robots.txt文件（这个文件是网站发出的声明，用于指导爬虫的行为，但只是声明、没有实际约束作用）。TEMPLATES_DIR ，用于指定startproject时的项目模板、genspider时的爬虫模板。这个文件夹默认是scrapy库路径下的templates目录。USER_AGENT="Scrapy/VERSION (+https://scrapy.org)" ，settings.py里设置UA的地方。但一般都是在下载中间件里设置，那里的设置会覆盖掉settings.py里的UA设置。（5）其他的不重要设置DEPTH_LIMIT=0 ，爬取网页的深度，0表示无限制。DEPTH_PRIORITY=0 ，用于调整不同深度的请求的优先级。优先级的算法为 request.priority = request.priority - ( depth * DEPTH_PRIORITY ) ，也就是 DEPTH_PRIORITY 为正时越深的请求优先级越低（广度优先遍历，BFO），负时相反（深度优先遍历，DFO）。scrapy默认是DFO，这在大多数情况下效率更高，如果你实在需要改成BFO，可以在settings.py中如下设置：DEPTH_PRIORITY = 1 SCHEDULER_DISK_QUEUE = 'scrapy.squeues.PickleFifoDiskQueue' SCHEDULER_MEMORY_QUEUE = 'scrapy.squeues.FifoMemoryQueue'DOWNLOAD_TIMEOUT=180 ，下载器的超时时间，单位为秒。还可以通过爬虫的 download_timeout 属性，和请求的 download_timeout 属性来指定。DOWNLOAD_MAXSIZE=1073741824 ，下载器最多下载的响应大小，单位为Byte，默认的大小为1024MB。同样，还能通过爬虫的 download_maxsize 或请求的 download_maxsize 属性指定。DOWNLOAD_WARNSIZE=33554432 ，当响应大小达到这个值后下载器会发出警告，默认值为32MB。爬虫、请求的 download_warnsize 属性也可以设置。JOBDIR='' ，指定一个路径，该路径用于保存爬虫状态。与爬虫的暂停、继续有关。REDIRECT_PRIORITY_ADJUST=+2 ，如果请求被重定向，这个选项用来设置重定向后的请求与原来请求的优先级关系，正表示重定向请求的优先级更高。URLLENGTH_LIMIT=2083 ，爬取的URL最大长度。这个是在发出请求的时候过滤url用的，因为有时网页服务器或者你自己的scrapy项目代码错误，会出现很长的url。如果想去掉这个限制，把它设置为0。
步骤/目录： 1.运行mysql数据库与准备工作 2.自建库 3.创建、修改、运行scrapy项目 （1）其他配置 （2）爬虫配置 a.a2u.py b.clarketm.py c.cool_proxy.py d.free_proxy_list.py e.ip3366.py f.ipaddress.py g.jiangxianli.py h.kuaidaili.py i.pubproxy.py j.rmccurdy.py k.sunny9577.py l.TheSpeedX.py （3）运行爬虫 4.验证ip 5.日常使用本文首发于个人博客https://lisper517.top/index.php/archives/48/，转载请注明出处。本文的目的是使用 scrapy+mysql 搭建一个自己的代理ip池，即ipool。由于本文是对前文进行阶段性总结+拓展，所以内容较多，笔者希望本文单独拿出来也能成功跟着一步步操作。本文写作日期为2022年9月11日。mysql数据库运行在树莓派上，其他部分运行在win10机器；使用的编辑器为VS Code。爬虫写的好，牢房进的早。使用爬虫时不要占用服务器太多流量，否则可能惹祸上身。笔者的能力有限，也并非CS从业人员，很多地方难免有纰漏或者不符合代码原则的地方，请在评论中指出。1.运行mysql数据库与准备工作笔者在树莓派上使用docker安装mysql，需要预先安装docker、docker-compose，在树莓派上安装docker、docker-compose可查看 Docker入门（一） 。也不一定非要在树莓派上运行mysql，或者也不一定要用docker，或者也可以用redis等其他数据库代替mysql（使用其他数据库时后续的py文件可能需要自行调整），反正最终的目的是运行一个数据库，可以参考 runoob教程 使用其他方法安装运行mysql数据库。在树莓派上进行如下操作：mkdir -p /docker/mysql/data/mysql mkdir -p /docker/mysql/conf/mysql mkdir -p /docker/mysql/backup/mysql docker pull mysql:8.0.29 nano /docker/mysql/docker-compose.yml在docker-compose.yml文件中写入如下内容：version: "3.9" services: mysql: image: mysql:8.0.29 environment: MYSQL_ROOT_PASSWORD: mysqlpasswd ports: - "53306:3306" command: - mysqld - --character-set-server=utf8mb4 volumes: - /docker/mysql/data/mysql:/var/lib/mysql - /docker/mysql/conf/mysql:/etc/mysql/conf.d - /docker/mysql/backup/mysql:/backup logging: driver: syslog restart: always记得把 MYSQL_ROOT_PASSWORD: mysqlpasswd 这里改成自己想设置的密码，把 53306 也可以改成自己想要的端口（不建议用默认的3306端口）。这里的 /docker/mysql/backup/mysql 目录是为了方便docker-mysql导出、备份数据。然后运行：cd /docker/mysql docker-compose config docker-compose up -d接下来进容器服务里看看mysql是否正常运行：docker exec -it mysql-mysql-1 bash #如果是老版本的docker-compose，把这一条命令中的所有 - 换成 _ mysql -uroot -p #输入密码 SHOW DATABASES;确认mysql正常运行即可。为了保证创建database和table的命令能记录下来，这些命令将写在python文件中。然后在win10机器上准备一下环境。需要安装python3，安装scrapy和其他库，创建自建库文件夹，创建scrapy项目文件夹。win10机器安装python3的过程这里就不讲了。安装好后，在cmd中运行 pip install Scrapy ，其他可能需要的库还有requests（处理网络请求）、bs4（对响应内容进行解析）、pymysql（连接mysql数据库）、lxml（网页解析器）、cryptography（连接mysql时密码加密）、selenium（操控浏览器），在cmd中使用 pip install requests bs4 pymysql lxml cryptography selenium 下载（这几个库在本文中不一定都会用到，但学习爬虫迟早会用），注意若pip下载失败就一直使用上述命令直到成功，或者bing搜索一下pip换源。最后创建文件夹 D:\PythonLib\MyPythonLib 和 D:\spiders\scrapy ，或者你也可以自己选择其他路径。2.自建库在 D:\PythonLib\MyPythonLib 中新建一个空白的 __init__.py 文件，然后到python的安装目录里存放第三方库的文件夹（win10上一般是site-packages目录），比如 C:\Program Files\Python310\Lib\site-packages 中，新建一个 PythonLib.pth 文件，写入一行内容： D:\PythonLib 。接下来创建自建库文件，本次需要用到的自建库主要与log、mysql相关。新建 D:\PythonLib\MyPythonLib\log.py ，写入：#!/usr/bin/python # -*- coding: UTF-8 -*- import logging, time, os def get_logger(abs_path: str, log_file_name: str = '', getLogger_name: str = ''): '''传入绝对路径和log文件名，返回log对象。 :param abs_path: log文件存储的绝对路径 :param log_file_name: log文件名，若log文件名为空，则使用默认的文件名 :param getLogger_name: logger对象名，如果在同一个程序中需要同时写多个log文件，建议这个参数不要相同 :return log对象''' try: formatter = logging.Formatter( '%(lineno)d : %(asctime)s : %(levelname)s : %(funcName)s : %(message)s' ) if log_file_name == '': log_file_name = 'log#{}.txt'.format(time.strftime('-%Y-%m-%d')) fileHandler = logging.FileHandler( (os.path.join(abs_path, log_file_name)), mode='w', encoding='utf-8') fileHandler.setFormatter(formatter) if getLogger_name == '': getLogger_name = 'logger' log = logging.getLogger(getLogger_name) log.setLevel(logging.DEBUG) log.addHandler(fileHandler) return log except: raise Exception('error: MyPythonLib.log.get_logger failed ')新建 D:\PythonLib\MyPythonLib\mysql.py ，写入（注意把get_conn函数中对应的host、port、密码改成自己的mysql配置）：#!/usr/bin/python # -*- coding: UTF-8 -*- import pymysql, os, logging, time from MyPythonLib import log as mylog def get_conn(database_name: str = 'mysql'): '''指定数据库名，返回mysql连接对象。 :param database_name: 数据库名，不指定时默认用mysql库 :return Connection''' try: conn = pymysql.connect(host='127.0.0.1', port=53306, user='root', passwd='mysqlpasswd', db=database_name, charset='utf8') return conn except: raise Exception('MyPythonLib.mysql.get_conn failed ') def str_into_mysql(string: str) -> str: '''输入一个字符串，将其转换为可以在mysql中储存的形式。 这个字符串的使用位置是列值处，比如 "SELECT * FROM 表名 WHERE 列名='字符串';" 中的字符串，若包含 \\ 、 ' 、 " 这三个字符，就需要用本函数处理。 :param string: 目标字符串 :return 可以存入mysql的字符串 ''' try: string = string.replace('\\', '\\\\') string = string.replace("'", "\\'") string = string.replace('"', '\\"') return string except: raise Exception('MyPythonLib.mysql.str_into_mysql failed ') def table_exists(cur, table_name: str) -> bool: '''判断表在当前库中是否存在。 :param cur: 游标对象 :param table_name: 表名 :return 存在则返回True，否则返回False ''' try: cur.execute("SHOW TABLES;") results = cur.fetchall() for table in results: table = table[0] if table == table_name: return True return False except: raise Exception('MyPythonLib.mysql.table_exists failed ') class mysql_api_ipool(): """通过这个接口实现提交爬取的ip、取出ip的操作。""" conn = get_conn() cur = conn.cursor() #这个cursor是类公用的，用于不需要实例化对象的mysql操作 def __init__(self, log: logging.Logger = None): """:param log: 可选的log对象，若不提供，将在当前文件夹下新建logs目录、创建log文件""" if log == None: log_dir = os.path.join(os.getcwd(), 'logs') if not os.path.isdir(log_dir): os.mkdir(log_dir) log_file_name = 'mysql_api_ipool_log#{}.txt'.format( time.strftime('-%Y-%m-%d_%H_%M_%S')) log = mylog.get_logger(log_dir, log_file_name, 'mysql_api_ipool') self.log = log message = 'mysql_api_ipool object initialized' self.log.info(message) self.cur = mysql_api_ipool.conn.cursor() #对象自己的cur mysql_api_ipool.check_table() message = 'database ipool and table ipool checked' self.log.info(message) def __del__(self): self.cur.close() message = 'mysql_api_ipool object deleted' self.log.info(message) @staticmethod def check_table(): """检查一下mysql中是否已经建好了表，若表不存在则新建表。 :这里并没有检查：若表存在，表的格式是否正确。 数据库的各列含义与默认值： :no，自增主键； :ip，代理ip（"0.0.0.0"）； :port，代理ip端口（0）； :anonymous，是否为高匿，0为否，1为是，3为其他（3）； :type，0为HTTP，1为HTTPS，2为都可，3为其他（3）； :physical_address，代理ip的地理位置（"unknown"）； :response_time，响应时间，单位为ms（-1）； :last_verify，上次验证该代理ip的时间（"1000-01-01 00:00:00"）； :grade，该代理ip的分数。每验证成功一次会加分，失败则扣分，扣到0会被删除，在爬取ip时不会使用到（100）； :created，该行创建时间。""" cur, conn = mysql_api_ipool.cur, mysql_api_ipool.conn command = ''' CREATE TABLE IF NOT EXISTS ipool ( no BIGINT UNSIGNED AUTO_INCREMENT, ip VARCHAR(50) NOT NULL DEFAULT "0.0.0.0" COMMENT "IP address", port SMALLINT UNSIGNED NOT NULL DEFAULT 0 COMMENT "port", anonymous BIT(2) NOT NULL DEFAULT 3 COMMENT "whether ip is anonymous", type BIT(2) NOT NULL DEFAULT 3 COMMENT "HTTP or HTTPS or both or others", physical_address VARCHAR(100) NOT NULL DEFAULT "unknown" COMMENT "where is the server", response_time MEDIUMINT NOT NULL DEFAULT -1 COMMENT "response_time in microseconds", last_verify DATETIME NOT NULL DEFAULT '1000-01-01 00:00:00' COMMENT "last verify time", grade TINYINT UNSIGNED NOT NULL DEFAULT 100 COMMENT "grade of ip, used for validation", created TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP, PRIMARY KEY (no));''' command = command.replace('\n', '') command = command.replace(' ', '') try: cur.execute("CREATE DATABASE IF NOT EXISTS ipool;") cur.execute("USE ipool;") cur.execute(command) conn.commit() except: conn.rollback() raise Exception( 'MyPythonLib.mysql.mysql_api_ipool.check_table failed') def save_ips(self, ips_dict_list: list): """存入多个ip（以字典的列表形式提交）。注意这里只是对不存在的数据添加默认值，没有验证数据类型等。 :param ips_dict_list: ip字典的列表，每个ip是一个字典。""" cur, conn, log = self.cur, mysql_api_ipool.conn, self.log if len(ips_dict_list) == 0: message = 'no ip submitted' log.info(message) return command = '''INSERT INTO ipool (ip,port,anonymous,type,physical_address,response_time,last_verify) VALUES ''' for ip_dict in ips_dict_list: ip, port, anonymous, type_, physical_address, response_time, last_verify = "0.0.0.0", 0, 3, 3, 'unknown', -1, "1000-01-01 00:00:00" if 'ip' in ip_dict.keys(): ip = ip_dict['ip'] if 'port' in ip_dict.keys(): port = ip_dict['port'] if 'anonymous' in ip_dict.keys(): anonymous = ip_dict['anonymous'] if 'type' in ip_dict.keys(): type_ = ip_dict['type'] if 'physical_address' in ip_dict.keys(): physical_address = ip_dict['physical_address'] physical_address = str_into_mysql(physical_address) if 'response_time' in ip_dict.keys(): response_time = ip_dict['response_time'] if 'last_verify' in ip_dict.keys(): last_verify = ip_dict['last_verify'] command += '("{}",{},{},{},"{}",{},"{}"),'.format( ip, port, anonymous, type_, physical_address, response_time, last_verify) message = 'trying to save ip:port={}:{}'.format(ip, port) log.info(message) command = command[:-1] + ';' try: cur.execute(command) conn.commit() message = 'saving ips successfully' log.info(message) except: conn.rollback() log.info('错误命令为：' + command) raise Exception( 'MyPythonLib.mysql.mysql_api_ipool.save_ips failed') def check_repeat(self): """检查爬到的数据里有没有重复的，只检查ip和port都相同的。 如果有重复的，保留no最小的一条记录。""" cur, conn, log = mysql_api_ipool.cur, mysql_api_ipool.conn, self.log cur.execute('USE ipool;') cur.execute( 'SELECT ip,port FROM ipool GROUP BY ip,port HAVING COUNT(*)>1;') repeated_items = cur.fetchall() if len(repeated_items) == 0: message = 'no repeated ip:port in ipool' log.info(message) return else: message = 'found repeated ip:port {} kinds'.format( len(repeated_items)) log.info(message) command = ''' DELETE FROM ipool WHERE no IN (SELECT no FROM (SELECT no FROM ipool WHERE (ip,port) IN (SELECT ip,port FROM ipool GROUP BY ip,port HAVING COUNT(*)>1) AND no NOT IN (SELECT MIN(no) FROM ipool GROUP BY ip,port HAVING COUNT(*)>1)) AS a);''' command = command.replace('\n', '') command = command.replace(' ', '') try: cur.execute(command) conn.commit() message = 'repeated ip:port deleted successfully. ' log.info(message) except: conn.rollback() raise Exception( 'MyPythonLib.mysql.mysql_api_ipool.check_repeat failed') @staticmethod def get_ips(random: bool = False, anonymous: list = [0, 1, 2, 3], type_: list = [0, 1, 2, 3], response_time: int = 30000, last_verify_interval: str = '48:00:00', grade: int = 100, limit: int = 1000) -> tuple: """根据要求返回多个ip，只返回ip和port。 :param random: 是否随机返回ip :param anonymous: 以列表形式指定匿名性 :param type_: 对HTTP还是HTTPS或其他代理 :param response_time: 响应时间在多少ms之内 :param last_verify_interval: 用于指定上次验证的时间距现在不超过多久，若为None，则是不限定验证时间 :param grade: ip的分数必须 >= grade :param limit: 最多返回几条代理ip :return tuple('ip:port','ip:port', ...)""" cur = mysql_api_ipool.cur cur.execute('USE ipool;') anonymous, type_ = tuple(anonymous), tuple(type_) if last_verify_interval == None: last_verify_interval = '99999999:00:00' command = ''' SELECT ip,port FROM ipool WHERE anonymous IN {} AND type IN {} AND response_time BETWEEN 0 AND {} AND (NOW()-last_verify)<"{}" AND grade >= {} ORDER BY response_time,last_verify DESC LIMIT {}; '''.format(anonymous, type_, response_time, last_verify_interval, grade, limit) if random: command = ''' SELECT ip,port FROM (SELECT ip,port FROM ipool WHERE anonymous IN {} AND type IN {} AND response_time BETWEEN 0 AND {} AND (NOW()-last_verify)<"{}" AND grade >= {} ) AS a ORDER BY RAND() LIMIT {}; '''.format(anonymous, type_, response_time, last_verify_interval, grade, limit) command = command.replace('\n', '') command = command.replace(' ', '') try: cur.execute(command) except: raise Exception('MyPythonLib.mysql.mysql_api_ipool.get_ips failed') ips = cur.fetchall() ips = tuple( [ips[i][0] + ':' + str(ips[i][1]) for i in range(len(ips))]) return ips def get_to_be_validated_ips(self, last_verify_interval: str = '48:00:00', limit: int = -1): """返回需要验证的ip，只返回no、ip和port。 :param last_verify_interval: 用于指定上次验证的时间距现在大于多久 :param limit: 最多返回几条代理ip，为-1时全部取出 :return tuple((no,ip,port,garde), (no,ip,port,garde), ...)""" cur, log = self.cur, self.log cur.execute('USE ipool;') command = ''' SELECT no,ip,port,grade FROM ipool WHERE (NOW()-last_verify)>"{}" ORDER BY last_verify,no LIMIT {}; '''.format(last_verify_interval, limit) command = command.replace('\n', '') command = command.replace(' ', '') if limit == -1: command = command.replace(' LIMIT -1', '') try: cur.execute(command) except: raise Exception( 'MyPythonLib.mysql.mysql_api_ipool.get_to_be_validated_ips failed' ) message = 'get {} to_be_validated ips'.format(limit) log.info(message) return cur.fetchall() def update_ip(self, ip_dict: dict): '''根据ip_dict更新数据，ip_dict中必须包含no、ip、port，另外还应该有至少一个需要更改的列，未提到的列不更改。''' cur, conn, log = self.cur, mysql_api_ipool.conn, self.log try: no, ip, port = ip_dict['no'], ip_dict['ip'], ip_dict['port'] except: raise Exception( 'MyPythonLib.mysql.mysql_api_ipool.update_ip failed: ip_dict must include no,ip,port' ) if len(ip_dict) <= 3: raise Exception( 'MyPythonLib.mysql.mysql_api_ipool.update_ip failed: at least 1 column must be updated' ) cur.execute('USE ipool;') command = 'SELECT no,ip,port FROM ipool WHERE no={};'.format(no) cur.execute(command) results = cur.fetchall() if len(results) == 0: message = 'ip no.{} not exists'.format(no) log.warning(message) return if ip != results[0][1] or port != results[0][2]: message = 'ip no.{} ip:port not right: {}:{}, recieved {}:{}, update given up'.format( no, results[0][1], results[0][2], ip, port) log.error(message) return command = '''UPDATE ipool SET ''' if 'anonymous' in ip_dict.keys(): command += 'anonymous={},'.format(ip_dict['anonymous']) if 'type' in ip_dict.keys(): command += 'type={},'.format(ip_dict['type']) if 'physical_address' in ip_dict.keys(): command += 'physical_address="{}",'.format( str_into_mysql(ip_dict['physical_address'])) if 'response_time' in ip_dict.keys(): command += 'response_time={},'.format(ip_dict['response_time']) if 'last_verify' in ip_dict.keys(): command += 'last_verify="{}",'.format(ip_dict['last_verify']) if 'grade' in ip_dict.keys(): command += 'grade={},'.format(ip_dict['grade']) command = command[:-1] + ' WHERE no={};'.format(no) try: cur.execute(command) conn.commit() message = 'update ip no.{} successfully'.format(no) log.info(message) except: conn.rollback() log.info('错误命令为：' + command) raise Exception( 'MyPythonLib.mysql.mysql_api_ipool.update_ip failed') 3.创建并修改scrapy项目本次使用到的提供代理ip的网站有12个，如下：http://www.ip3366.net/free/ https://www.kuaidaili.com https://ip.jiangxianli.com/ https://www.cool-proxy.net/ https://free-proxy-list.net https://www.ipaddress.com/proxy-list/ https://www.proxynova.com/proxy-server-list/ http://pubproxy.com/api/proxy?limit=5&format=txt&type=http&level=anonymous&last_check=60&no_country=CN https://www.rmccurdy.com/.scripts/proxy/good.txt https://raw.githubusercontent.com/sunny9577/proxy-scraper/master/proxies.json https://raw.githubusercontent.com/a2u/free-proxy-list/master/free-proxy-list.txt https://raw.githubusercontent.com/clarketm/proxy-list/master/proxy-list.txt https://raw.githubusercontent.com/TheSpeedX/SOCKS-List/master/http.txt其中 https://www.kuaidaili.com 页数很多（最早的数据是2014年的，有4000多页），但用scrapy会封ip，所以只爬前2页。另外，还有3个可提供代理ip的网站：http://proxy-list.org/english/ https://www.proxynova.com/proxy-server-list/ http://spys.one/en/anonymous-proxy-list/这三个网站使用js生成代理ip表格，爬取它们最简单的方法是使用selenium，但它们提供的代理ip不多，且proxynova可用 https://raw.githubusercontent.com/sunny9577/proxy-scraper/master/proxies.json 替代，所以这里就不爬了。（1）其他配置使用cmd，进行如下操作：D: cd D:\spiders\scrapy scrapy startproject ipool在资源管理器中打开 D:\spiders\scrapy\ipool\ipool ，修改一下4个文件，修改settings.py为：# Scrapy settings for ipool project # # For simplicity, this file contains only settings considered important or # commonly used. You can find more settings consulting the documentation: # # https://docs.scrapy.org/en/latest/topics/settings.html # https://docs.scrapy.org/en/latest/topics/downloader-middleware.html # https://docs.scrapy.org/en/latest/topics/spider-middleware.html LOG_LEVEL = 'ERROR' # 只打印错误信息 BOT_NAME = 'ipool' SPIDER_MODULES = ['ipool.spiders'] NEWSPIDER_MODULE = 'ipool.spiders' # Crawl responsibly by identifying yourself (and your website) on the user-agent #USER_AGENT = 'ipool (+http://www.yourdomain.com)' # Obey robots.txt rules ROBOTSTXT_OBEY = False #不遵守robots.txt # Configure maximum concurrent requests performed by Scrapy (default: 16) #CONCURRENT_REQUESTS = 32 # Configure a delay for requests for the same website (default: 0) # See https://docs.scrapy.org/en/latest/topics/settings.html#download-delay # See also autothrottle settings and docs #DOWNLOAD_DELAY = 3 # The download delay setting will honor only one of: #CONCURRENT_REQUESTS_PER_DOMAIN = 16 #CONCURRENT_REQUESTS_PER_IP = 16 # Disable cookies (enabled by default) #COOKIES_ENABLED = False # Disable Telnet Console (enabled by default) #TELNETCONSOLE_ENABLED = False # Override the default request headers: #DEFAULT_REQUEST_HEADERS = { # 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', # 'Accept-Language': 'en', #} # Enable or disable spider middlewares # See https://docs.scrapy.org/en/latest/topics/spider-middleware.html #SPIDER_MIDDLEWARES = { # 'ipool.middlewares.IpoolSpiderMiddleware': 543, #} # Enable or disable downloader middlewares # See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html DOWNLOADER_MIDDLEWARES = { 'ipool.middlewares.IpoolDownloaderMiddleware': 543, } # Enable or disable extensions # See https://docs.scrapy.org/en/latest/topics/extensions.html #EXTENSIONS = { # 'scrapy.extensions.telnet.TelnetConsole': None, #} # Configure item pipelines # See https://docs.scrapy.org/en/latest/topics/item-pipeline.html ITEM_PIPELINES = {'ipool.pipelines.MysqlPipeline': 1} # Enable and configure the AutoThrottle extension (disabled by default) # See https://docs.scrapy.org/en/latest/topics/autothrottle.html #AUTOTHROTTLE_ENABLED = True # The initial download delay #AUTOTHROTTLE_START_DELAY = 5 # The maximum download delay to be set in case of high latencies #AUTOTHROTTLE_MAX_DELAY = 60 # The average number of requests Scrapy should be sending in parallel to # each remote server #AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0 # Enable showing throttling stats for every response received: #AUTOTHROTTLE_DEBUG = False # Enable and configure HTTP caching (disabled by default) # See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html#httpcache-middleware-settings #HTTPCACHE_ENABLED = True #HTTPCACHE_EXPIRATION_SECS = 0 #HTTPCACHE_DIR = 'httpcache' #HTTPCACHE_IGNORE_HTTP_CODES = [] #HTTPCACHE_STORAGE = 'scrapy.extensions.httpcache.FilesystemCacheStorage' pipelines.py为：# Define your item pipelines here # # Don't forget to add your pipeline to the ITEM_PIPELINES setting # See: https://docs.scrapy.org/en/latest/topics/item-pipeline.html # useful for handling different item types with a single interface from itemadapter import ItemAdapter from MyPythonLib.mysql import mysql_api_ipool import os class MysqlPipeline: def open_spider(self, spider): self.mysql_api_ipool = mysql_api_ipool() def process_item(self, item, spider): self.mysql_api_ipool.save_ips(item['ips_dict_list']) return item def close_spider(self, spider): self.mysql_api_ipool.check_repeat()middlewares.py为：# Define here the models for your spider middleware # # See documentation in: # https://docs.scrapy.org/en/latest/topics/spider-middleware.html from scrapy import signals # useful for handling different item types with a single interface from itemadapter import is_item, ItemAdapter # 爬虫中间件 class IpoolSpiderMiddleware: # Not all methods need to be defined. If a method is not defined, # scrapy acts as if the spider middleware does not modify the # passed objects. @classmethod def from_crawler(cls, crawler): # This method is used by Scrapy to create your spiders. s = cls() crawler.signals.connect(s.spider_opened, signal=signals.spider_opened) return s def process_spider_input(self, response, spider): # Called for each response that goes through the spider # middleware and into the spider. # Should return None or raise an exception. return None def process_spider_output(self, response, result, spider): # Called with the results returned from the Spider, after # it has processed the response. # Must return an iterable of Request, or item objects. for i in result: yield i def process_spider_exception(self, response, exception, spider): # Called when a spider or process_spider_input() method # (from other spider middleware) raises an exception. # Should return either None or an iterable of Request or item objects. pass def process_start_requests(self, start_requests, spider): # Called with the start requests of the spider, and works # similarly to the process_spider_output() method, except # that it doesn’t have a response associated. # Must return only requests (not items). for r in start_requests: yield r def spider_opened(self, spider): spider.logger.info('Spider opened: %s' % spider.name) import random # 下载中间件 class IpoolDownloaderMiddleware: # Not all methods need to be defined. If a method is not defined, # scrapy acts as if the downloader middleware does not modify the # passed objects. UA_list = [ 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36', 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36 Edg/105.0.1343.33', 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:104.0) Gecko/20100101 Firefox/104.0' ] @classmethod def from_crawler(cls, crawler): # This method is used by Scrapy to create your spiders. s = cls() crawler.signals.connect(s.spider_opened, signal=signals.spider_opened) return s # 拦截无异常的请求，可进行UA伪装 def process_request(self, request, spider): # Called for each request that goes through the downloader # middleware. # Must either: # - return None: continue processing this request # - or return a Response object # - or return a Request object # - or raise IgnoreRequest: process_exception() methods of # installed downloader middleware will be called request.headers['User-Agent'] = random.choice(self.UA_list) return None # 拦截所有响应，可用于爬动态加载的数据 def process_response(self, request, response, spider): # Called with the response returned from the downloader. # Must either; # - return a Response object # - return a Request object # - or raise IgnoreRequest return response # 拦截发生异常的请求，可进行代理ip设置 def process_exception(self, request, exception, spider): # Called when a download handler or a process_request() # (from other downloader middleware) raises an exception. # Must either: # - return None: continue processing this exception # - return a Response object: stops process_exception() chain # - return a Request object: stops process_exception() chain pass def spider_opened(self, spider): spider.logger.info('Spider opened: %s' % spider.name) items.py为：# Define here the models for your scraped items # # See documentation in: # https://docs.scrapy.org/en/latest/topics/items.html import scrapy class IpoolItem(scrapy.Item): # define the fields for your item here like: # name = scrapy.Field() ips_dict_list = scrapy.Field() （2）爬虫配置要爬12个网站，所以需要写12个爬虫。其实有几个网站的数据格式差不多，这些爬虫还能再精简合并一下。这些爬虫都放在 D:\spiders\scrapy\ipool\ipool\spiders 目录下。a.a2u.pyimport scrapy, re from ipool.items import IpoolItem class A2uSpider(scrapy.Spider): name = 'a2u' start_urls = [ 'https://raw.githubusercontent.com/a2u/free-proxy-list/master/free-proxy-list.txt' ] def parse(self, response): ip_nodes = re.findall( r'[1-2]{0,1}\d{0,2}\.[1-2]{0,1}\d{0,2}\.[1-2]{0,1}\d{0,2}\.[1-2]{0,1}\d{0,2}:\d{1,5}', response.text) ips_dict_list = [] #这是一个字典的列表，和mysql_api.save_ips里的格式匹配 for ip_node in ip_nodes: ip_dict = {} ip, port = ip_node.split(':') port = int(port) ip_dict['ip'] = ip ip_dict['port'] = port ips_dict_list.append(ip_dict) item = IpoolItem() item['ips_dict_list'] = ips_dict_list yield itemb.clarketm.pyimport scrapy, re from ipool.items import IpoolItem class ClarketmSpider(scrapy.Spider): name = 'clarketm' start_urls = [ 'https://raw.githubusercontent.com/clarketm/proxy-list/master/proxy-list.txt' ] def parse(self, response): ip_nodes = re.findall( r'[1-2]{0,1}\d{0,2}\.[1-2]{0,1}\d{0,2}\.[1-2]{0,1}\d{0,2}\.[1-2]{0,1}\d{0,2}:\d{1,5}', response.text) ips_dict_list = [] #这是一个字典的列表，和mysql_api.save_ips里的格式匹配 for ip_node in ip_nodes: ip_dict = {} ip, port = ip_node.split(':') ip = ip.strip() port = int(port.strip()) ip_dict['ip'] = ip ip_dict['port'] = port ips_dict_list.append(ip_dict) item = IpoolItem() item['ips_dict_list'] = ips_dict_list yield item c.cool_proxy.pyimport scrapy, json, time from ipool.items import IpoolItem class CoolProxySpider(scrapy.Spider): name = 'cool_proxy' start_urls = ['https://www.cool-proxy.net/proxies.json'] def parse(self, response): ip_nodes = json.loads(response.text.replace('\'', '\"')) ips_dict_list = [] #这是一个字典的列表，和mysql_api.save_ips里的格式匹配 for ip_node in ip_nodes: ip_dict = {} ip = ip_node['ip'].strip() port = int(ip_node['port']) anonymous = (0, 1)[1 == ip_node['anonymous']] type_ = 3 physical_address = ip_node['country_name'].strip() response_time = int(1000 * float(ip_node['response_time_average'])) last_verify = float(ip_node['update_time']) last_verify = time.localtime(last_verify) last_verify = time.strftime('%Y-%m-%d %H:%M:%S', last_verify) ip_dict['ip'] = ip ip_dict['port'] = port ip_dict['anonymous'] = anonymous ip_dict['type'] = type_ ip_dict['physical_address'] = physical_address ip_dict['response_time'] = response_time ip_dict['last_verify'] = last_verify ips_dict_list.append(ip_dict) item = IpoolItem() item['ips_dict_list'] = ips_dict_list yield item d.free_proxy_list.pyimport scrapy, time from ipool.items import IpoolItem class FreeProxyListSpider(scrapy.Spider): name = 'free_proxy_list' start_urls = ['https://free-proxy-list.net/'] def parse(self, response): ip_nodes = response.xpath( r'//*[@id="list"]/div/div[2]/div/table/tbody/tr') ips_dict_list = [] #这是一个字典的列表，和mysql_api.save_ips里的格式匹配 for ip_node in ip_nodes: ip_dict = {} ip = ip_node.xpath('./td[1]/text()').extract()[0].strip() port = int(ip_node.xpath('./td[2]/text()').extract()[0].strip()) anonymous = ( 0, 1)['anonymous' in ip_node.xpath('./td[5]/text()').extract()[0]] # (a,b)[条件] 也是python三元表达式的一种写法。注意条件为真时，表达式整体值为b type_ = (0, 1)['yes' in ip_node.xpath('./td[7]/text()').extract()[0]] physical_address = ip_node.xpath( './td[4]/text()').extract()[0].strip() response_time = 1000 last_verify = time.strftime('%Y-%m-%d %H:%M:%S') ip_dict['ip'] = ip ip_dict['port'] = port ip_dict['anonymous'] = anonymous ip_dict['type'] = type_ ip_dict['physical_address'] = physical_address ip_dict['response_time'] = response_time ip_dict['last_verify'] = last_verify ips_dict_list.append(ip_dict) item = IpoolItem() item['ips_dict_list'] = ips_dict_list yield iteme.ip3366.pyimport scrapy from ipool.items import IpoolItem class Ip3366Spider(scrapy.Spider): name = 'ip3366' #allowed_domains = ['www.ip3366.net'] # 允许的域名。如果域名在此之外，则不会爬取 start_urls = [ 'http://www.ip3366.net/free/', 'http://www.ip3366.net/free/?stype=2' ] # 起始url的列表 base_url = 'http://www.ip3366.net/free/' def parse(self, response): ip_nodes = response.xpath(r'//*[@id="list"]/table/tbody/tr') ips_dict_list = [] #这是一个字典的列表，和mysql_api.save_ips里的格式匹配 for ip_node in ip_nodes: ip_dict = {} ip = ip_node.xpath(r'./td[1]/text()').extract()[0].strip() port = int(ip_node.xpath(r'./td[2]/text()').extract()[0].strip()) anonymous = ( 0, 1)['高匿' in ip_node.xpath(r'./td[3]/text()').extract()[0]] # (a,b)[条件] 也是python三元表达式的一种写法。注意条件为真时，表达式整体值为b type_ = ( 0, 1)['HTTPS' in ip_node.xpath(r'./td[4]/text()').extract()[0]] physical_address = ip_node.xpath( r'./td[5]/text()').extract()[0].strip() response_time = 1000 * int( ip_node.xpath(r'./td[6]/text()').extract()[0].strip().replace( '秒', '')) response_time = (response_time, 250)[response_time == 0] #这个网站中有很多代理ip的延迟为0，不太准确，一律改成250即0.25s last_verify = ip_node.xpath( r'./td[7]/text()').extract()[0].strip().replace('/', '-') ip_dict['ip'] = ip ip_dict['port'] = port ip_dict['anonymous'] = anonymous ip_dict['type'] = type_ ip_dict['physical_address'] = physical_address ip_dict['response_time'] = response_time ip_dict['last_verify'] = last_verify ips_dict_list.append(ip_dict) item = IpoolItem() item['ips_dict_list'] = ips_dict_list yield item next_url = response.xpath( r'//*[@id="listnav"]/ul/a[text()="下一页"]/@href').extract() if next_url != None and len(next_url) != 0: next_url = Ip3366Spider.base_url + next_url[0] yield scrapy.Request(url=next_url, callback=self.parse) # 回调函数是自己，即递归解析网页 # 本方法在爬虫结束时被调用 def closed(self, spider): passf.ipaddress.pyimport scrapy from ipool.items import IpoolItem class IpaddressSpider(scrapy.Spider): name = 'ipaddress' start_urls = ['https://www.ipaddress.com/proxy-list/'] def parse(self, response): ip_nodes = response.xpath(r'/html/body/div[1]/main/table/tbody/tr') ips_dict_list = [] #这是一个字典的列表，和mysql_api.save_ips里的格式匹配 for ip_node in ip_nodes: ip_dict = {} ip = ip_node.xpath('./td[1]/a[1]/text()').extract()[0].strip() port = int( ip_node.xpath('./td[1]/text()').extract()[0].strip().replace( '"', '').replace(':', '')) anonymous = ( 0, 1)['anonymous' in ip_node.xpath('./td[2]/text()').extract()[0]] # (a,b)[条件] 也是python三元表达式的一种写法。注意条件为真时，表达式整体值为b type_ = 3 physical_address = ip_node.xpath( './td[3]/text()').extract()[0].strip()[3:] response_time = 1000 last_verify = ip_node.xpath( './td[4]/text()').extract()[0].strip() + ':00' ip_dict['ip'] = ip ip_dict['port'] = port ip_dict['anonymous'] = anonymous ip_dict['type'] = type_ ip_dict['physical_address'] = physical_address ip_dict['response_time'] = response_time ip_dict['last_verify'] = last_verify ips_dict_list.append(ip_dict) item = IpoolItem() item['ips_dict_list'] = ips_dict_list yield itemg.jiangxianli.pyimport scrapy, json from ipool.items import IpoolItem class JiangxianliSpider(scrapy.Spider): name = 'jiangxianli' start_urls = ['https://ip.jiangxianli.com/api/proxy_ips?page=1'] def parse(self, response): ip_nodes = json.loads(response.text)['data']['data'] ips_dict_list = [] #这是一个字典的列表，和mysql_api.save_ips里的格式匹配 for i in range(len(ip_nodes)): ip_dict = {} ip = ip_nodes[i]['ip'].strip() port = int(ip_nodes[i]['port'].strip()) anonymous = (0, 1)[2 == ip_nodes[i]['anonymity']] type_ = (0, 1)['https' == ip_nodes[i]['protocol']] physical_address = ip_nodes[i]['ip_address'].strip() response_time = ip_nodes[i]['speed'] last_verify = ip_nodes[i]['validated_at'].strip() ip_dict['ip'] = ip ip_dict['port'] = port ip_dict['anonymous'] = anonymous ip_dict['type'] = type_ ip_dict['physical_address'] = physical_address ip_dict['response_time'] = response_time ip_dict['last_verify'] = last_verify ips_dict_list.append(ip_dict) item = IpoolItem() item['ips_dict_list'] = ips_dict_list yield item next_url = json.loads(response.text)['data']['next_page_url'] if next_url != None: yield scrapy.Request(url=next_url, callback=self.parse) h.kuaidaili.pyimport scrapy from ipool.items import IpoolItem class KuaidailiSpider(scrapy.Spider): name = 'kuaidaili' #allowed_domains = ['www.kuaidaili.com'] start_urls = [ 'https://free.kuaidaili.com/free/inha/1/', 'https://free.kuaidaili.com/free/intr/1/' ] base_url = 'https://free.kuaidaili.com' custom_settings = { 'CONCURRENT_REQUESTS': 1, 'DOWNLOAD_DELAY': 10, 'CONCURRENT_REQUESTS_PER_DOMAIN': 1 } def parse(self, response): ip_nodes = response.xpath(r'//*[@id="list"]/table/tbody/tr') ips_dict_list = [] #这是一个字典的列表，和mysql_api.save_ips里的格式匹配 for ip_node in ip_nodes: ip_dict = {} ip = ip_node.xpath(r'./td[1]/text()').extract()[0].strip() port = int(ip_node.xpath(r'./td[2]/text()').extract()[0].strip()) anonymous = ( 0, 1)['高匿' in ip_node.xpath(r'./td[3]/text()').extract()[0]] # (a,b)[条件] 也是python三元表达式的一种写法。注意条件为真时，表达式整体值为b type_ = ( 0, 1)['HTTPS' in ip_node.xpath(r'./td[4]/text()').extract()[0]] physical_address = ip_node.xpath( r'./td[5]/text()').extract()[0].strip() response_time = 1000 * float( ip_node.xpath(r'./td[6]/text()').extract()[0].strip().replace( '秒', '')) last_verify = ip_node.xpath(r'./td[7]/text()').extract()[0].strip() ip_dict['ip'] = ip ip_dict['port'] = port ip_dict['anonymous'] = anonymous ip_dict['type'] = type_ ip_dict['physical_address'] = physical_address ip_dict['response_time'] = response_time ip_dict['last_verify'] = last_verify ips_dict_list.append(ip_dict) item = IpoolItem() item['ips_dict_list'] = ips_dict_list yield item page_num = response.url.split('/')[-2] next_url = response.url.replace(page_num, str(int(page_num) + 1)) other_urls_list = response.xpath( r'//*[@id="listnav"]/ul/li/a/@href').extract() other_urls_list = [ KuaidailiSpider.base_url + other_urls_list[i] for i in range(len(other_urls_list)) ] if int(page_num) < 5 and next_url in other_urls_list: yield scrapy.Request(url=next_url, callback=self.parse) custom_settings 用于指定这个爬虫的个性化设置，本例中限制了爬取的频率；并且只爬前5页。i.pubproxy.pyimport scrapy, re from ipool.items import IpoolItem class PubproxySpider(scrapy.Spider): name = 'pubproxy' start_urls = [ 'http://pubproxy.com/api/proxy?limit=5&format=txt&type=http&level=anonymous&last_check=60&no_country=CN' ] def parse(self, response): ip_nodes = re.findall( r'[1-2]{0,1}\d{0,2}\.[1-2]{0,1}\d{0,2}\.[1-2]{0,1}\d{0,2}\.[1-2]{0,1}\d{0,2}:\d{1,5}', response.text) ips_dict_list = [] #这是一个字典的列表，和mysql_api.save_ips里的格式匹配 for ip_node in ip_nodes: ip_dict = {} ip, port = ip_node.split(':') port = int(port) ip_dict['ip'] = ip ip_dict['port'] = port ips_dict_list.append(ip_dict) item = IpoolItem() item['ips_dict_list'] = ips_dict_list yield itemj.rmccurdy.pyimport scrapy, re from ipool.items import IpoolItem class RmccurdySpider(scrapy.Spider): name = 'rmccurdy' start_urls = ['https://www.rmccurdy.com/.scripts/proxy/good.txt'] def parse(self, response): ip_nodes = re.findall( r'[1-2]{0,1}\d{0,2}\.[1-2]{0,1}\d{0,2}\.[1-2]{0,1}\d{0,2}\.[1-2]{0,1}\d{0,2}:\d{1,5}', response.text) ips_dict_list = [] #这是一个字典的列表，和mysql_api.save_ips里的格式匹配 for ip_node in ip_nodes: ip_dict = {} ip, port = ip_node.split(':') port = int(port) ip_dict['ip'] = ip ip_dict['port'] = port ips_dict_list.append(ip_dict) item = IpoolItem() item['ips_dict_list'] = ips_dict_list yield item k.sunny9577.pyimport scrapy, json from ipool.items import IpoolItem class Sunny9577Spider(scrapy.Spider): name = 'sunny9577' start_urls = [ 'https://raw.githubusercontent.com/sunny9577/proxy-scraper/master/proxies.json' ] def parse(self, response): ip_nodes = json.loads(response.text)['proxynova'] ips_dict_list = [] #这是一个字典的列表，和mysql_api.save_ips里的格式匹配 for ip_node in ip_nodes: ip_dict = {} ip = ip_node['ip'].strip() port = int(ip_node['port'].strip()) anonymous = (0, 1)['Anonymous' in ip_node['anonymity']] # (a,b)[条件] 也是python三元表达式的一种写法。注意条件为真时，表达式整体值为b type_ = 3 physical_address = ip_node['country'] ip_dict['ip'] = ip ip_dict['port'] = port ip_dict['anonymous'] = anonymous ip_dict['type'] = type_ ip_dict['physical_address'] = physical_address ips_dict_list.append(ip_dict) item = IpoolItem() item['ips_dict_list'] = ips_dict_list yield item l.TheSpeedX.pyimport scrapy, re from ipool.items import IpoolItem class ThespeedxSpider(scrapy.Spider): name = 'TheSpeedX' start_urls = [ 'https://raw.githubusercontent.com/TheSpeedX/SOCKS-List/master/http.txt', 'https://raw.githubusercontent.com/TheSpeedX/SOCKS-List/master/socks5.txt', 'https://raw.githubusercontent.com/TheSpeedX/SOCKS-List/master/socks4.txt' ] def parse(self, response): ip_nodes = re.findall( r'[1-2]{0,1}\d{0,2}\.[1-2]{0,1}\d{0,2}\.[1-2]{0,1}\d{0,2}\.[1-2]{0,1}\d{0,2}:\d{1,5}', response.text) ips_dict_list = [] #这是一个字典的列表，和mysql_api.save_ips里的格式匹配 for ip_node in ip_nodes: ip_dict = {} ip, port = ip_node.split(':') ip = ip.strip() port = int(port.strip()) ip_dict['ip'] = ip ip_dict['port'] = port ips_dict_list.append(ip_dict) item = IpoolItem() item['ips_dict_list'] = ips_dict_list yield item （3）运行爬虫这里涉及到了多个爬虫，运行的时候一个个使用 scrapy crawl 爬虫名 比较麻烦。scrapy提供了其他方法，可同时运行一个项目中的多个爬虫或全部爬虫。以下内容参考了 这篇文章 。在 D:\spiders\scrapy\ipool\ipool 下新建一个py文件，比如 crawl_all.py ，内容为：from scrapy.crawler import CrawlerProcess from scrapy.utils.project import get_project_settings from scrapy.spiderloader import SpiderLoader # 根据项目配置获取 CrawlerProcess 实例 process = CrawlerProcess(get_project_settings()) # 获取 spiderloader 对象，以进一步获取项目下所有爬虫名称 spider_loader = SpiderLoader(get_project_settings()) # 添加全部爬虫 for spidername in spider_loader.list(): process.crawl(spidername) # 添加一个爬虫 #process.crawl('爬虫名') # 执行 process.start()如上所述，添加一个爬虫就用 process.crawl('爬虫名') ，而 spider_loader.list() 方法可获取项目中的所有爬虫名，两者搭配就可运行全部爬虫。另外需要提醒，笔者爬取的几个网站中有几个不能直接访问，比如 raw.githubusercontent.com 那几个，你可以把它们去掉，或者使用一些设置。如果需要使用V2rayN并使用requests库或scrapy，可查看 Docker实践——如何查阅wiki百科及V2Ray ，其中提到了如何使用V2rayN并运行爬虫。最后，在cmd中进入 D:\spiders\scrapy\ipool\ipool ，运行 python crawl_all.py 即可。4.验证ip原本想使用树莓派多进程验证ip，但占用树莓派资源太高，改在win10电脑上验证。在win10电脑的自建库目录中，除了 log.py 、 mysql.py ，还需要添加一个spider.py，文件内容如下：#!/usr/bin/python # -*- coding: UTF-8 -*- import requests, random headers = { 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36' } def get_session(max_retries: int = 3) -> requests.Session: '''接受最大重连数，返回一个requests会话对象 :param max_retries: 最大重连次数 :return 会话对象''' # 设置重连次数。包括开头，若出现异常，最多尝试连接4次 try: session = requests.Session() session.mount('http://', requests.adapters.HTTPAdapter(max_retries=max_retries)) session.mount('https://', requests.adapters.HTTPAdapter(max_retries=max_retries)) return session except: raise Exception('error: MyPythonLib.spider.get_session failed ') def multiprocess_job_spliter(process_num: int, to_do_list, random_shuffle: bool = True) -> dict: '''给出进程数、总任务序列，返回分好的任务列表。 :param process_num: 进程数（需要将总任务序列分为多少份） :param to_do_list: 总任务序列（元组或列表），比如待爬取的所有url字符串 :param random_shuffle: 是否打乱总任务序列的顺序 :return {0: todo_list1, 1: todo_list2, ... }''' try: to_do_list = list(to_do_list) if random_shuffle: random.shuffle(to_do_list) total_num = len(to_do_list) one_share = total_num // process_num share_list = {} for i in range(0, process_num): if i != process_num - 1: share_list[i] = to_do_list[(i * one_share):((i + 1) * one_share)] elif i == process_num - 1: share_list[i] = to_do_list[(i * one_share):] return share_list except: raise Exception( 'error: MyPythonLib.spider.multiprocess_job_spliter failed ')这个文件后面还有很多内容，但实际上只会用到 multiprocess_job_spliter 这个函数。然后在一个合适的地方放python脚本，比如就放在scrapy的ipool项目中： D:\spiders\scrapy\ipool\ipool\update_ips.py ，文件内容如下：#!/usr/bin/python # -*- coding: UTF-8 -*- from MyPythonLib import mysql, spider from MyPythonLib import log as mylog import logging, os, time, multiprocessing headers = spider.headers class update_ips(): test_http_url = 'http://example.org/' test_https_url = 'https://example.org/' def __init__(self, log: logging.Logger = None): """:param log: 可选的log对象，若不提供，将在当前文件夹下新建logs目录、创建log文件""" if log == None: log_dir = os.path.join(os.getcwd(), 'logs') if not os.path.isdir(log_dir): os.mkdir(log_dir) log_file_name = 'mysql_api_ipool_log#{}.txt'.format( time.strftime('-%Y-%m-%d_%H_%M_%S')) log = mylog.get_logger(log_dir, log_file_name, 'mysql_api_ipool') self.log = log message = 'ready to update ips' self.log.info(message) self.mysql_api_ipool = mysql.mysql_api_ipool(self.log) self.conn = self.mysql_api_ipool.conn self.cur = self.conn.cursor() self.mysql_api_ipool.check_table() message = 'database ipool and table ipool checked' self.log.info(message) self.cur.execute('USE ipool;') self.conn.commit() def __del__(self): self.cur.close() message = 'update ips complete' self.log.info(message) @staticmethod def validate_ips_single_process(ips_list: list, process_no: int = 0, grade_minus: int = 50): '''验证ip，只涉及type、last_verify、grade的改变。若grade<=0，会将这个ip删除。 :这个方法是单进程的。 :param ips_list: 待验证ip的列表，格式为 [(no,ip,port,grade), (no,ip,port,grade), ...] :param process_no: 进程编号，主要用于log文件取名 :param grade_minus: 若代理ip无法使用，则扣除该分数''' global headers log = mylog.get_logger( abs_path=os.path.join(os.getcwd(), 'logs'), log_file_name='validate_ips-{}'.format(process_no), getLogger_name=str(process_no)) message = 'received {} ips, validating...'.format(len(ips_list)) log.info(message) mysql_api_ipool = mysql.mysql_api_ipool(log) session = spider.get_session(max_retries=1) for no, ip, port, grade in ips_list: ip_port = '{}:{}'.format(ip, port) proxies = {'http': ip_port, 'https': ip_port} http_success, https_success = True, True try: session.get(url=update_ips.test_http_url, timeout=10, headers=headers, proxies=proxies) except: http_success = False try: session.get(url=update_ips.test_https_url, timeout=10, headers=headers, proxies=proxies) except: https_success = False if http_success and not https_success: type_ = 0 elif https_success and not http_success: type_ = 1 elif http_success and https_success: type_ = 2 elif not http_success and not https_success: type_ = 3 ip_dict = {} ip_dict['no'] = no ip_dict['ip'] = ip ip_dict['port'] = port ip_dict['type'] = type_ ip_dict['last_verify'] = time.strftime('%Y-%m-%d %H:%M:%S') if type_ == 3: grade -= grade_minus else: grade = 100 ip_dict['grade'] = grade mysql_api_ipool.update_ip(ip_dict=ip_dict) def validate_ips_multiprocess(self, process_num: int = 5, limit: int = -1, grade_minus: int = 50): '''验证ip，只涉及type、last_verify、grade的改变。若grade<=0，会将这个ip删除。 :这个方法是多进程的，将会调用 validate_ips_single_process 。 :param process_num: 验证ip的进程数 :param limit: 限制待验证ip的条数，为-1时表示全部取出 :param grade_minus: 若代理ip无法使用，则扣除该分数''' cur, conn, log = self.cur, self.conn, self.log command = 'DELETE FROM ipool WHERE grade<=0;' try: cur.execute(command) conn.commit() message = 'delete ips where grade<=0 successfully' log.info(message) except: conn.rollback() log.info('错误命令为：' + command) raise Exception( 'update_ips.validate_ips_multiprocess failed: delete error') ips_list = self.mysql_api_ipool.get_to_be_validated_ips(limit=limit) ips_list_dict = spider.multiprocess_job_spliter(process_num, ips_list) process_pool = multiprocessing.Pool(processes=process_num) for i in range(process_num): process_pool.apply_async(update_ips.validate_ips_single_process, args=(ips_list_dict[i], i, grade_minus)) process_pool.close() process_pool.join() try: cur.execute(command) conn.commit() message = 'delete ips where grade<=0 successfully' log.info(message) except: conn.rollback() log.info('错误命令为：' + command) raise Exception( 'update_ips.validate_ips_multiprocess failed: delete error') if __name__ == '__main__': update_ips().validate_ips_multiprocess(process_num=60, limit=-1, grade_minus=50) update_ips().validate_ips_multiprocess(process_num=60, limit=-1, grade_minus=50) 中， process_num 和 limit 可以自行修改，主要与win10机器的配置有关，验证ip的过程mysql读写频率不算高。另外，这里的验证ip只涉及了type（http还是https）、last_verify（最后验证时间）、grade（ip分数）的改变，如果想一同更新代理ip的匿名性、物理地址、响应时间，可以在这里的基础上自行更改。最后，在cmd中，使用：cd D:\spiders\scrapy\ipool\ipool python update_ips.py即可运行验证ip的py脚本。5.日常使用爬取ip和验证ip的脚本可以每天运行一次，顺序如下：（1）若需要使用V2rayN，先开启V2rayN。（2）在cmd中使用如下命令：cd D:\spiders\scrapy\ipool\ipool python crawl_all.py python update_ips.py即可。如果想把命令写成脚本、开机自动运行，可以参考 不同系统下的开机自动运行 。另外，mysql记得时常备份。使用代理ip时，用 MyPythonLib.mysql.mysql_api_ipool.get_ips 方法可取出ip。
步骤/目录： 1.应用场景与scrapy架构 2.安装scrapy与创建项目 3.使用scrapy爬取http://www.ip3366.net/free/ （1）创建项目与修改框架 （2）自建库文件的内容 （3）运行爬虫本文首发于个人博客https://lisper517.top/index.php/archives/47/，转载请注明出处。本文的目的是讲解scrapy的使用方法。本文写作日期为2022年9月10日。爬虫写的好，牢房进的早。使用爬虫时不要占用服务器太多流量，否则可能惹祸上身。笔者的能力有限，也并非CS从业人员，很多地方难免有纰漏或者不符合代码原则的地方，请在评论中指出。1.应用场景与scrapy架构在之前 Python，爬虫与深度学习（6）——爬虫详解（二）构建自己的ip池（简易版） 这篇文章中，提到过 https://free.kuaidaili.com/free/ 这个网站，它列出了4700多页免费代理ip。爬取这些ip是比较耗时的，但还算比较简单，因为需要爬取的每个网页的网址，区别只在最后的页码数字不同。如果我遇到一个网站，我想爬取它的全站数据，也就是说从它的根页面开始，把根页面需要的数据爬取下来，然后提取出根页面上的所有符合条件的超链接（仅提取属于这个网站的，比如下面有备案信息的网页会提供一个到工信部备案的链接，这种链接就不要），依次访问这些链接，对这些链接也都重复 爬取数据、提取链接 这两步，这样一直重复下去，就能爬取该站所有在页面上出现过的链接（有些链接可能在所有页面上都没出现过，这种隐藏起来的链接就没办法了），也就是一般意义上的爬取全站数据。但是要注意，有些时候网页会故意放一些正常人无法看到的链接（在浏览器页面上无法显示的链接，比如超出显示范围，或者没有写内容的a标签等），爬虫通过阅读源码能轻易找到，但是如果访问到这个页面，网页服务器就会识别到你是爬虫，不再提供服务，这也是一种反爬方法。作为写爬虫的一方，我们能做到的是把提取链接的规则写小一点，少用正则提取页面上的所有链接。scrapy就是一种用于全站爬取的爬虫框架，关于它组件的介绍见 runoob教程 ，这里简单介绍一下。用scrapy爬取一个网站的流程是，从一个初始url开始，下载器（Downloader）通过url获得响应，拿给爬虫（Spider）解析、提取出信息和其他要访问的url，把信息交给管道（Item Pipeline）进行存储，比如存成txt文件或存入数据库；其他url则在爬虫中以请求的形式送入调度器（Scheduler），等待合适的时机交给下载器继续下载。另外还有下载中间件和Spider中间件，用于自定义配置，比如有时你想用selenium获得请求，那么就在中间件里把获得响应的部分进行修改。另外，4个组件的中心是scrapy引擎，这个核心组件负责调度、传递组件间的信息。最后，scrapy爬取网页是采用多进程的方法，所以爬起来非常快，但是也要注意可能被封ip。所以在使用scrapy之前，最好手上已经有一定的代理ip。2.安装scrapy与创建项目使用 pip install Scrapy 安装scrapy，然后在cmd或终端中，进入放scrapy爬虫项目的文件夹（这个文件夹以后可以专门用来放爬虫），创建项目：scrapy startproject 爬虫项目名然后进入项目文件夹，创建爬虫：scrapy genspider 爬虫名 "爬虫要爬的网址"网址可以随便写，文件里可以修改。另外，一个项目里可以有多个爬虫。对项目里的各个文件进行修改，最后使用 scrapy crawl 爬虫名 运行爬虫。下面就展示一下如何修改这些文件。3.使用scrapy爬取http://www.ip3366.net/free/这里以 http://www.ip3366.net/free/ 为例，展示一下scrapy的使用，需要爬取该网站的ip。以下的命令在win10机器上运行。（1）创建项目与修改框架新建一个文件夹，比如 D:\spiders\scrapy ，然后进行如下操作：scrapy startproject ipool cd ipool scrapy genspider ip3366 http://www.ip3366.net/free/在创建过程中，可以看到scrapy有框架模板、爬虫模板，目前使用的是默认模板。以后对爬虫若有一定要求，可编写自己的模板。然后在资源管理器中进入 D:\spiders\scrapy\ipool\ipool ，可以看到items、middlewares、pipelines、settings，和ip3366这五个py文件，接下来修改这几个文件。items.py原内容如下：# Define here the models for your scraped items # # See documentation in: # https://docs.scrapy.org/en/latest/topics/items.html import scrapy class IpoolItem(scrapy.Item): # define the fields for your item here like: # name = scrapy.Field() pass这个文件的修改很简单，它用于定义爬取字段。将items.py修改如下：# Define here the models for your scraped items # # See documentation in: # https://docs.scrapy.org/en/latest/topics/items.html import scrapy class IpoolItem(scrapy.Item): # define the fields for your item here like: # name = scrapy.Field() ips_dict_list = scrapy.Field() 然后是middlewares.py，就是两个中间件的配置文件，可以进行一些自定义设置，本例中只是添加一下UA。原文件如下：# Define here the models for your spider middleware # # See documentation in: # https://docs.scrapy.org/en/latest/topics/spider-middleware.html from scrapy import signals # useful for handling different item types with a single interface from itemadapter import is_item, ItemAdapter class IpoolSpiderMiddleware: # Not all methods need to be defined. If a method is not defined, # scrapy acts as if the spider middleware does not modify the # passed objects. @classmethod def from_crawler(cls, crawler): # This method is used by Scrapy to create your spiders. s = cls() crawler.signals.connect(s.spider_opened, signal=signals.spider_opened) return s def process_spider_input(self, response, spider): # Called for each response that goes through the spider # middleware and into the spider. # Should return None or raise an exception. return None def process_spider_output(self, response, result, spider): # Called with the results returned from the Spider, after # it has processed the response. # Must return an iterable of Request, or item objects. for i in result: yield i def process_spider_exception(self, response, exception, spider): # Called when a spider or process_spider_input() method # (from other spider middleware) raises an exception. # Should return either None or an iterable of Request or item objects. pass def process_start_requests(self, start_requests, spider): # Called with the start requests of the spider, and works # similarly to the process_spider_output() method, except # that it doesn’t have a response associated. # Must return only requests (not items). for r in start_requests: yield r def spider_opened(self, spider): spider.logger.info('Spider opened: %s' % spider.name) class IpoolDownloaderMiddleware: # Not all methods need to be defined. If a method is not defined, # scrapy acts as if the downloader middleware does not modify the # passed objects. @classmethod def from_crawler(cls, crawler): # This method is used by Scrapy to create your spiders. s = cls() crawler.signals.connect(s.spider_opened, signal=signals.spider_opened) return s def process_request(self, request, spider): # Called for each request that goes through the downloader # middleware. # Must either: # - return None: continue processing this request # - or return a Response object # - or return a Request object # - or raise IgnoreRequest: process_exception() methods of # installed downloader middleware will be called return None def process_response(self, request, response, spider): # Called with the response returned from the downloader. # Must either; # - return a Response object # - return a Request object # - or raise IgnoreRequest return response def process_exception(self, request, exception, spider): # Called when a download handler or a process_request() # (from other downloader middleware) raises an exception. # Must either: # - return None: continue processing this exception # - return a Response object: stops process_exception() chain # - return a Request object: stops process_exception() chain pass def spider_opened(self, spider): spider.logger.info('Spider opened: %s' % spider.name) 在下载中间件里对UA进行设置，另外笔者添加了一些注释：# Define here the models for your spider middleware # # See documentation in: # https://docs.scrapy.org/en/latest/topics/spider-middleware.html from scrapy import signals # useful for handling different item types with a single interface from itemadapter import is_item, ItemAdapter # 爬虫中间件 class IpoolSpiderMiddleware: # Not all methods need to be defined. If a method is not defined, # scrapy acts as if the spider middleware does not modify the # passed objects. @classmethod def from_crawler(cls, crawler): # This method is used by Scrapy to create your spiders. s = cls() crawler.signals.connect(s.spider_opened, signal=signals.spider_opened) return s def process_spider_input(self, response, spider): # Called for each response that goes through the spider # middleware and into the spider. # Should return None or raise an exception. return None def process_spider_output(self, response, result, spider): # Called with the results returned from the Spider, after # it has processed the response. # Must return an iterable of Request, or item objects. for i in result: yield i def process_spider_exception(self, response, exception, spider): # Called when a spider or process_spider_input() method # (from other spider middleware) raises an exception. # Should return either None or an iterable of Request or item objects. pass def process_start_requests(self, start_requests, spider): # Called with the start requests of the spider, and works # similarly to the process_spider_output() method, except # that it doesn’t have a response associated. # Must return only requests (not items). for r in start_requests: yield r def spider_opened(self, spider): spider.logger.info('Spider opened: %s' % spider.name) import random # 下载中间件 class IpoolDownloaderMiddleware: # Not all methods need to be defined. If a method is not defined, # scrapy acts as if the downloader middleware does not modify the # passed objects. UA_list = [ 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36', 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36 Edg/105.0.1343.33', 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:104.0) Gecko/20100101 Firefox/104.0' ] @classmethod def from_crawler(cls, crawler): # This method is used by Scrapy to create your spiders. s = cls() crawler.signals.connect(s.spider_opened, signal=signals.spider_opened) return s # 拦截无异常的请求，可进行UA伪装 def process_request(self, request, spider): # Called for each request that goes through the downloader # middleware. # Must either: # - return None: continue processing this request # - or return a Response object # - or return a Request object # - or raise IgnoreRequest: process_exception() methods of # installed downloader middleware will be called request.headers['User-Agent'] = random.choice(self.UA_list) return None # 拦截所有响应，可用于爬动态加载的数据 def process_response(self, request, response, spider): # Called with the response returned from the downloader. # Must either; # - return a Response object # - return a Request object # - or raise IgnoreRequest return response # 拦截发生异常的请求，可进行代理ip设置 def process_exception(self, request, exception, spider): # Called when a download handler or a process_request() # (from other downloader middleware) raises an exception. # Must either: # - return None: continue processing this exception # - return a Response object: stops process_exception() chain # - return a Request object: stops process_exception() chain pass def spider_opened(self, spider): spider.logger.info('Spider opened: %s' % spider.name) 接着是pipeline.py，在这里可以处理爬到的数据。管道类可以有多个，最后在setting.py里确定优先级。另外，一定要确定每个pipeline最后都return item，这样一个pipeline把spider传过来的数据（就是item）处理完后才能传给下一个优先级的pipeline。其原文件为：# Define your item pipelines here # # Don't forget to add your pipeline to the ITEM_PIPELINES setting # See: https://docs.scrapy.org/en/latest/topics/item-pipeline.html # useful for handling different item types with a single interface from itemadapter import ItemAdapter class IpoolPipeline: def process_item(self, item, spider): return item 改写为：# Define your item pipelines here # # Don't forget to add your pipeline to the ITEM_PIPELINES setting # See: https://docs.scrapy.org/en/latest/topics/item-pipeline.html # useful for handling different item types with a single interface from itemadapter import ItemAdapter from MyPythonLib.ipool_mysql_api import mysql_api import os class IpoolPipeline: #这个方法在父类中定义过，其只在爬虫开始时调用一次 def open_spider(self, spider): self.fp = open(os.path.join(os.getcwd(), 'ips.txt'), 'w', encoding='utf-8') #这个方法用于处理数据 def process_item(self, item, spider): for ip_dict in item['ips_dict_list']: ip = ip_dict['ip'] port = ip_dict['port'] self.fp.write(ip + ':' + str(port) + '\n') return item #这个方法在父类中定义过，其只在爬虫结束时调用一次 def close_spider(self, spider): self.fp.close() class MysqlPipeline: def open_spider(self, spider): self.mysql_api = mysql_api() def process_item(self, item, spider): self.mysql_api.save_ips(item['ips_dict_list']) return item def close_spider(self, spider): self.mysql_api.check_repeat()在这里定义了两个管道类，一个把爬到的ip、port存成txt文件，另一个把数据存入mysql。这里的MyPythonLib是笔者的自建库，其文件内容将在下一节展示。接下来是setting.py，其原文件内容如下：# Scrapy settings for ipool project # # For simplicity, this file contains only settings considered important or # commonly used. You can find more settings consulting the documentation: # # https://docs.scrapy.org/en/latest/topics/settings.html # https://docs.scrapy.org/en/latest/topics/downloader-middleware.html # https://docs.scrapy.org/en/latest/topics/spider-middleware.html BOT_NAME = 'ipool' SPIDER_MODULES = ['ipool.spiders'] NEWSPIDER_MODULE = 'ipool.spiders' # Crawl responsibly by identifying yourself (and your website) on the user-agent #USER_AGENT = 'ipool (+http://www.yourdomain.com)' # Obey robots.txt rules ROBOTSTXT_OBEY = True # Configure maximum concurrent requests performed by Scrapy (default: 16) #CONCURRENT_REQUESTS = 32 # Configure a delay for requests for the same website (default: 0) # See https://docs.scrapy.org/en/latest/topics/settings.html#download-delay # See also autothrottle settings and docs #DOWNLOAD_DELAY = 3 # The download delay setting will honor only one of: #CONCURRENT_REQUESTS_PER_DOMAIN = 16 #CONCURRENT_REQUESTS_PER_IP = 16 # Disable cookies (enabled by default) #COOKIES_ENABLED = False # Disable Telnet Console (enabled by default) #TELNETCONSOLE_ENABLED = False # Override the default request headers: #DEFAULT_REQUEST_HEADERS = { # 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', # 'Accept-Language': 'en', #} # Enable or disable spider middlewares # See https://docs.scrapy.org/en/latest/topics/spider-middleware.html #SPIDER_MIDDLEWARES = { # 'ipool.middlewares.IpoolSpiderMiddleware': 543, #} # Enable or disable downloader middlewares # See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html #DOWNLOADER_MIDDLEWARES = { # 'ipool.middlewares.IpoolDownloaderMiddleware': 543, #} # Enable or disable extensions # See https://docs.scrapy.org/en/latest/topics/extensions.html #EXTENSIONS = { # 'scrapy.extensions.telnet.TelnetConsole': None, #} # Configure item pipelines # See https://docs.scrapy.org/en/latest/topics/item-pipeline.html #ITEM_PIPELINES = { # 'ipool.pipelines.IpoolPipeline': 300, #} # Enable and configure the AutoThrottle extension (disabled by default) # See https://docs.scrapy.org/en/latest/topics/autothrottle.html #AUTOTHROTTLE_ENABLED = True # The initial download delay #AUTOTHROTTLE_START_DELAY = 5 # The maximum download delay to be set in case of high latencies #AUTOTHROTTLE_MAX_DELAY = 60 # The average number of requests Scrapy should be sending in parallel to # each remote server #AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0 # Enable showing throttling stats for every response received: #AUTOTHROTTLE_DEBUG = False # Enable and configure HTTP caching (disabled by default) # See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html#httpcache-middleware-settings #HTTPCACHE_ENABLED = True #HTTPCACHE_EXPIRATION_SECS = 0 #HTTPCACHE_DIR = 'httpcache' #HTTPCACHE_IGNORE_HTTP_CODES = [] #HTTPCACHE_STORAGE = 'scrapy.extensions.httpcache.FilesystemCacheStorage' 在这里添加、修改几行：# Scrapy settings for ipool project # # For simplicity, this file contains only settings considered important or # commonly used. You can find more settings consulting the documentation: # # https://docs.scrapy.org/en/latest/topics/settings.html # https://docs.scrapy.org/en/latest/topics/downloader-middleware.html # https://docs.scrapy.org/en/latest/topics/spider-middleware.html LOG_LEVEL = 'ERROR' # 只打印错误信息 BOT_NAME = 'ipool' SPIDER_MODULES = ['ipool.spiders'] NEWSPIDER_MODULE = 'ipool.spiders' # Crawl responsibly by identifying yourself (and your website) on the user-agent #USER_AGENT = 'ipool (+http://www.yourdomain.com)' # Obey robots.txt rules ROBOTSTXT_OBEY = False #不遵守robots.txt # Configure maximum concurrent requests performed by Scrapy (default: 16) #CONCURRENT_REQUESTS = 32 # Configure a delay for requests for the same website (default: 0) # See https://docs.scrapy.org/en/latest/topics/settings.html#download-delay # See also autothrottle settings and docs #DOWNLOAD_DELAY = 3 # The download delay setting will honor only one of: #CONCURRENT_REQUESTS_PER_DOMAIN = 16 #CONCURRENT_REQUESTS_PER_IP = 16 # Disable cookies (enabled by default) #COOKIES_ENABLED = False # Disable Telnet Console (enabled by default) #TELNETCONSOLE_ENABLED = False # Override the default request headers: #DEFAULT_REQUEST_HEADERS = { # 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', # 'Accept-Language': 'en', #} # Enable or disable spider middlewares # See https://docs.scrapy.org/en/latest/topics/spider-middleware.html #SPIDER_MIDDLEWARES = { # 'ipool.middlewares.IpoolSpiderMiddleware': 543, #} # Enable or disable downloader middlewares # See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html DOWNLOADER_MIDDLEWARES = { 'ipool.middlewares.IpoolDownloaderMiddleware': 543, } # Enable or disable extensions # See https://docs.scrapy.org/en/latest/topics/extensions.html #EXTENSIONS = { # 'scrapy.extensions.telnet.TelnetConsole': None, #} # Configure item pipelines # See https://docs.scrapy.org/en/latest/topics/item-pipeline.html ITEM_PIPELINES = { 'ipool.pipelines.IpoolPipeline': 2, #item数据按对应数字从小到大的顺序流经管道 'ipool.pipelines.MysqlPipeline': 1 } # Enable and configure the AutoThrottle extension (disabled by default) # See https://docs.scrapy.org/en/latest/topics/autothrottle.html #AUTOTHROTTLE_ENABLED = True # The initial download delay #AUTOTHROTTLE_START_DELAY = 5 # The maximum download delay to be set in case of high latencies #AUTOTHROTTLE_MAX_DELAY = 60 # The average number of requests Scrapy should be sending in parallel to # each remote server #AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0 # Enable showing throttling stats for every response received: #AUTOTHROTTLE_DEBUG = False # Enable and configure HTTP caching (disabled by default) # See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html#httpcache-middleware-settings #HTTPCACHE_ENABLED = True #HTTPCACHE_EXPIRATION_SECS = 0 #HTTPCACHE_DIR = 'httpcache' #HTTPCACHE_IGNORE_HTTP_CODES = [] #HTTPCACHE_STORAGE = 'scrapy.extensions.httpcache.FilesystemCacheStorage' 最后是ip3366.py，这个文件主要用于解析爬到的内容。原文件如下：import scrapy class Ip3366Spider(scrapy.Spider): name = 'ip3366' allowed_domains = ['www.ip3366.net'] start_urls = ['http://www.ip3366.net/'] def parse(self, response): pass 把它修改为：import scrapy from ipool.items import IpoolItem class Ip3366Spider(scrapy.Spider): name = 'ip3366' #allowed_domains = ['www.ip3366.net'] # 允许的域名。如果域名在此之外，则不会爬取 start_urls = [ 'http://www.ip3366.net/free/', 'http://www.ip3366.net/free/?stype=2' ] # 起始url的列表 base_url = 'http://www.ip3366.net/free/' def parse(self, response): ip_nodes = response.xpath(r'//*[@id="list"]/table/tbody/tr') ips_dict_list = [] #这是一个字典的列表，和mysql_api.save_ips里的格式匹配 for ip_node in ip_nodes: ip_dict = {} ip = ip_node.xpath(r'./td[1]/text()').extract()[0].strip() port = int(ip_node.xpath(r'./td[2]/text()').extract()[0].strip()) anonymous = ( 0, 1)['高匿' in ip_node.xpath(r'./td[3]/text()').extract()[0]] # (a,b)[条件] 也是python三元表达式的一种写法。注意条件为真时，表达式整体值为b type_ = ( 0, 1)['HTTPS' in ip_node.xpath(r'./td[4]/text()').extract()[0]] physical_address = ip_node.xpath( r'./td[5]/text()').extract()[0].strip() response_time = 1000 * int( ip_node.xpath(r'./td[6]/text()').extract()[0].strip().replace( '秒', '')) response_time = (response_time, 250)[response_time == 0] #这个网站中有很多代理ip的延迟为0，不太准确，一律改成250即0.25s last_verify = ip_node.xpath( r'./td[7]/text()').extract()[0].strip().replace('/', '-') ip_dict['ip'] = ip ip_dict['port'] = port ip_dict['anonymous'] = anonymous ip_dict['type'] = type_ ip_dict['physical_address'] = physical_address ip_dict['response_time'] = response_time ip_dict['last_verify'] = last_verify ips_dict_list.append(ip_dict) item = IpoolItem() item['ips_dict_list'] = ips_dict_list yield item next_url = response.xpath( r'//*[@id="listnav"]/ul/a[text()="下一页"]/@href').extract() if next_url != None and len(next_url) != 0: next_url = Ip3366Spider.base_url + next_url[0] yield scrapy.Request(url=next_url, callback=self.parse) # 回调函数是自己，即递归解析网页 # 本方法在爬虫结束时被调用 def closed(self, spider): pass（2）自建库文件的内容创建自建库的方法见 Python，爬虫与深度学习（7）——建立自己的库 。笔者使用的ipool_mysql_api.py文件内容为：#!/usr/bin/python # -*- coding: UTF-8 -*- import os, logging, time from MyPythonLib import mysql from MyPythonLib import log as mylog if not os.path.isdir(os.path.join(os.getcwd(), 'logs')): os.mkdir(os.path.join(os.getcwd(), 'logs')) class mysql_api(): """通过这个接口实现提交爬取的ip、取出ip的操作。""" conn = mysql.get_conn() cur = conn.cursor() #这个cur是类公用的，用于干一些不需要实例化对象的工作 def __init__(self, log: logging.Logger = None): if log == None: log = mylog.get_logger( os.path.join(os.getcwd(), 'logs'), 'mysql_api_log#{}.txt'.format( time.strftime('-%Y-%m-%d_%H_%M_%S')), 'mysql_api') self.log = log message = 'one of mysql_api object initialized' self.log.info(message) self.cur = mysql_api.conn.cursor() mysql_api.check_table() message = 'database ipool and table ipool checked' self.log.info(message) def __del__(self): self.cur.close() message = 'one of mysql_api object deleted' self.log.info(message) @staticmethod def check_table(): """检查一下mysql中是否已经建好了表，若表不存在则新建表。 这里并没有检查：若表存在，表的格式是否正确。 :数据库的各列含义与默认值： :no，自增主键； :ip，代理ip（"0.0.0.0"）； :port，代理ip端口（0）； :anonymous，是否为高匿，0为否，1为是，3为其他（3）； :type，0为HTTP，1为HTTPS，3为其他（3）； :physical_address，代理ip的地理位置（"unknown"）； :response_time，响应时间，单位为ms（-1）； :last_verify，上次验证该代理ip的时间（"1000-01-01 00:00:00"）； :created，该项目创建时间。""" cur, conn = mysql_api.cur, mysql_api.conn command = ''' CREATE TABLE IF NOT EXISTS ipool ( no BIGINT UNSIGNED AUTO_INCREMENT, ip VARCHAR(50) NOT NULL DEFAULT "0.0.0.0" COMMENT "IP address", port SMALLINT UNSIGNED NOT NULL DEFAULT 0 COMMENT "port", anonymous BIT(2) NOT NULL DEFAULT 3 COMMENT "whether ip is anonymous", type BIT(2) NOT NULL DEFAULT 3 COMMENT "HTTP or HTTPS or both or others", physical_address VARCHAR(100) NOT NULL DEFAULT "unknown" COMMENT "where is the server", response_time MEDIUMINT NOT NULL DEFAULT -1 COMMENT "response_time in microseconds", last_verify DATETIME NOT NULL DEFAULT '1000-01-01 00:00:00' COMMENT "last verify time", created TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP, PRIMARY KEY (no));''' command = command.replace('\n', '') command = command.replace(' ', '') try: cur.execute("CREATE DATABASE IF NOT EXISTS ipool;") cur.execute("USE ipool;") cur.execute(command) conn.commit() except: conn.rollback() raise Exception('mysql_api.check_table failed') def save_ips(self, ips_list: list): """存入多个ip（以字典的列表形式提交）。注意这里只是对不存在的数据添加默认值，没有验证数据类型等。 :数据库的各列含义与默认值： :no，自增主键； :ip，代理ip（"0.0.0.0"）； :port，代理ip端口（0）； :anonymous，是否为高匿，0为否，1为是，3为其他（3）； :type，0为HTTP，1为HTTPS，3为其他（3）； :physical_address，代理ip的地理位置（"unknown"）； :response_time，响应时间，单位为ms（-1）； :last_verify，上次验证该代理ip的时间（"1000-01-01 00:00:00"）； :created，该项目创建时间。""" cur, conn, log = self.cur, mysql_api.conn, self.log command = '''INSERT INTO ipool (ip,port,anonymous,type,physical_address,response_time,last_verify) VALUES ''' if len(ips_list) == 0: message = 'no ip from this website' log.info(message) return for ip_dict in ips_list: ip, port, anonymous, type_, physical_address, response_time, last_verify = "0.0.0.0", 0, 3, 3, 'unknown', -1, "1000-01-01 00:00:00" if 'ip' in ip_dict.keys(): ip = ip_dict['ip'] if 'port' in ip_dict.keys(): port = ip_dict['port'] if 'anonymous' in ip_dict.keys(): anonymous = ip_dict['anonymous'] if 'type' in ip_dict.keys(): type_ = ip_dict['type'] if 'physical_address' in ip_dict.keys(): physical_address = ip_dict['physical_address'] if 'response_time' in ip_dict.keys(): response_time = ip_dict['response_time'] if 'last_verify' in ip_dict.keys(): last_verify = ip_dict['last_verify'] command += '("{}",{},{},{},"{}",{},"{}"),'.format( ip, port, anonymous, type_, physical_address, response_time, last_verify) message = 'trying to save ip:port={}:{}'.format(ip, port) log.info(message) command = command[:-1] + ';' try: cur.execute(command) conn.commit() message = 'saving ips successfully' log.info(message) except: conn.rollback() log.info('错误命令为：' + command) raise Exception('mysql_api.save_ips failed') def check_repeat(self): """检查爬到的数据里有没有重复的，只检查ip和port都相同的。如果有重复的，保留最新的一条记录，也就是no最大的。""" cur, conn, log = mysql_api.cur, mysql_api.conn, self.log cur.execute('USE ipool;') cur.execute( 'SELECT ip,port FROM ipool GROUP BY ip,port HAVING COUNT(*)>1;') repeated_items = cur.fetchall() if len(repeated_items) == 0: message = 'no repeated ip:port in ipool' log.info(message) return else: message = 'found repeated ip:port {} kinds'.format( len(repeated_items)) log.info(message) command = ''' DELETE FROM ipool WHERE no IN (SELECT no FROM (SELECT no FROM ipool WHERE (ip,port) IN (SELECT ip,port FROM ipool GROUP BY ip,port HAVING COUNT(*)>1) AND no NOT IN (SELECT MAX(no) FROM ipool GROUP BY ip,port HAVING COUNT(*)>1)) AS a);''' command = command.replace('\n', '') command = command.replace(' ', '') try: cur.execute(command) conn.commit() message = 'repeated ip:port deleted successfully. ' log.info(message) except: conn.rollback() raise Exception('mysql_api.check_repeat failed') @staticmethod def get_ips(random: bool = False, anonymous: list = [0, 1, 2, 3], type_: list = [0, 1, 2, 3], response_time: int = 30000, last_verify_interval: str = '48:00:00', limit: int = 1000) -> tuple: """根据要求返回多个ip，只返回ip和port。 :param random: 是否随机返回ip :param anonymous: 以列表形式指定匿名性 :param type_: 对HTTP还是HTTPS或其他代理 :param response_time: 响应时间在多少ms之内 :param last_verify_interval: 用于指定上次验证的时间距现在不超过多久 :param limit: 最多返回几条代理ip :return tuple('ip:port','ip:port', ...)""" cur = mysql_api.cur cur.execute('USE ipool;') anonymous, type_ = tuple(anonymous), tuple(type_) command = ''' SELECT ip,port FROM ipool WHERE anonymous IN {} AND type IN {} AND response_time BETWEEN 0 AND {} AND (NOW()-last_verify)<"{}" ORDER BY response_time,last_verify DESC LIMIT {}; '''.format(anonymous, type_, response_time, last_verify_interval, limit) if random: command = ''' SELECT ip,port FROM (SELECT ip,port FROM ipool WHERE anonymous IN {} AND type IN {} AND response_time BETWEEN 0 AND {} AND (NOW()-last_verify)<"{}" ) ORDER BY RAND() LIMIT {}; '''.format(anonymous, type_, response_time, last_verify_interval, limit) command = command.replace('\n', '') command = command.replace(' ', '') try: cur.execute(command) except: raise Exception('mysql_api.get_ips failed') ips = cur.fetchall() ips = tuple( [ips[i][0] + ':' + str(ips[i][1]) for i in range(len(ips))]) return ipsmysql和log这两个自建库的内容见 Python，爬虫与深度学习（7）——建立自己的库 。（3）运行爬虫使用cmd进入 D:\spiders\scrapy\ipool\ipool ，运行 scrapy crawl ip3366 即可开启爬虫。
步骤/目录： 1.应用场景 2.准备工作 3.在个人库中添加selenium相关代码 4.selenium个人库使用范例本文首发于个人博客https://lisper517.top/index.php/archives/46/，转载请注明出处。本文的目的是讲解selenium，并在个人库中添加相应代码。本文写作日期为2022年9月10日。笔者的能力有限，也并非CS从业人员，很多地方难免有纰漏或者不符合代码原则的地方，请在评论中指出。1.应用场景在爬虫泛滥的今天，各网站采用了越发先进的方法识别爬虫。如果不是经验丰富的爬虫老手，总会在爬某个网站时翻车（老手也是从翻车一步步过来的）。如果对爬取速度要求不是太高，就可以使用selenium与实际的浏览器搭配爬取网页。目前很多网站使用了JavaScript在客户端生成页面，selenium的方法就十分适合爬取这些网页；如果无论怎么办都会被识别，笔者能提供的最终解决方案是手动开启浏览器并交给selenium控制。selenium由于要操作浏览器而速度稍慢，这是其唯一的缺点。2.准备工作下载selenium：pip install selenium另外，edge刚推出时，老版本的selenium还需要下msedge-selenium-tools来帮助扩展edge功能，现在已经不用了。如果新版selenium用不了edge的话，使用 pip install msedge-selenium-tools 来安装这个扩展包，并且selenium也会回到老版本。下载浏览器：去官网下载chrome、edge、firefox浏览器，注意不要下到别人暗改的版本了。老的爬虫教程可能会推荐下载phantomjs浏览器，这是一个没有界面的浏览器，现在很多网站都对phantomjs浏览器有一些识别与限制，所以已经不建议再使用了，何况chrome、edge、firefox也支持无界面。最后是下载webdriver，即浏览器驱动，注意的是和浏览器版本匹配。有些时候浏览器升级了，也要及时更换驱动。chrome、edge、firefox的驱动下载网址各自为：https://chromedriver.chromium.org/downloads https://developer.microsoft.com/en-us/microsoft-edge/tools/webdriver/ https://github.com/mozilla/geckodriver/releases以chrome为例，打开浏览器，在网址栏输入 chrome://version/ ，可以看到版本（还有 可执行文件路径、个人资料路径 ，下面可能会用到），根据版本选择对应的chrome驱动即可。最后，记下 可执行文件路径 ，把三个浏览器的驱动都放在一个文件夹中，比如 D:\webdrivers ，然后在个人库中添加selenium相关的代码。笔者下面将给出在win10机器上selenium的一种使用方法，可根据自身情况对代码进行修改。除了自己对selenium二次开发，还可以看看别人的成品，比如helium。3.在个人库中添加selenium相关代码添加一个driver类，需要实现的功能有：根据网页返回网页源码，和返回某个标签对象。from logging import Logger import selenium, os from selenium.webdriver.common.by import By from selenium.webdriver.support.ui import WebDriverWait from selenium.webdriver.support import expected_conditions as EC from selenium.webdriver import Chrome from selenium.webdriver.chrome.options import Options as ChromeOptions # chrome from selenium.webdriver import Edge from selenium.webdriver.edge.options import Options as EdgeOptions #from msedge.selenium_tools import Edge, EdgeOptions # edge(支持新版edge需要安扩展包pip install msedge-selenium-tools) from selenium.webdriver import Firefox from selenium.webdriver.firefox.options import Options as FirefoxOptions # firefox class driver(): '''用selenium提供的浏览器类。 driver().help() 可打印帮助。''' binary_location_chrome = r'C:\Program Files\Google\Chrome\Application\chrome.exe' binary_location_edge = r'C:\Program Files (x86)\Microsoft\Edge\Application\msedge.exe' binary_location_firefox = r'C:\Program Files\Mozilla Firefox\firefox.exe' #浏览器可执行文件的路径 driver_path = r'D:\webdrivers' #浏览器驱动的位置 driver_download_url = '''webdriver下载地址： https://chromedriver.chromium.org/downloads https://developer.microsoft.com/en-us/microsoft-edge/tools/webdriver/ https://github.com/mozilla/geckodriver/releases ''' userdata_dir_chrome = os.path.join(driver_path, 'userdata_chrome') if not os.path.isdir(userdata_dir_chrome): os.mkdir(userdata_dir_chrome) userdata_dir_edge = os.path.join(driver_path, 'userdata_edge') if not os.path.isdir(userdata_dir_edge): os.mkdir(userdata_dir_edge) userdata_dir_firefox = os.path.join(driver_path, 'userdata_firefox') if not os.path.isdir(userdata_dir_firefox): os.mkdir(userdata_dir_firefox) def __init__(self, kind: str = 'chrome', options=None, headless=True, disguise=True, independent_userdata=True, log: Logger = False) -> None: ''':param kind: 指定浏览器的种类，支持 edge/chrome/firefox :param options: 是 EdgeOptions 或 ChromeOptions 或 FirefoxOptions ，与kind对应 :param headless: 浏览器是否无头（不显示界面） :param disguise: 是否伪装成正常浏览器（仅chrome和edge） :param independent_userdata: 是否使用独立的userdata。如果开启，该文件夹会在driver_path路径下 :param log: 可选的log对象，用于输出日志''' self.kind = kind self.headless = headless self.disguise = disguise self.independent_userdata = independent_userdata self.options = (options, self.get_default_options())[options == None] #这里没有检查options和kind是否对应 self.browser = self.get_browser() self.log = log def __del__(self): self.browser.quit() @staticmethod def get_options_obj(kind: str, headless: bool = True, disguise: bool = True, independent_userdata: bool = False, userdata_dir: str = ''): '''根据浏览器种类，返回一个初始的options对象。 :注意这个方法可静态调用，你可以用这个方法得到初始options、自己加其他参数。 :param kind: 指定浏览器的种类，支持 edge/chrome/firefox :param headless: 浏览器是否无头（不显示界面） :param disguise: 是否伪装成正常浏览器（仅chrome和edge） :param independent_userdata: 是否使用独立的userdata。如果开启，需要指定userdata的路径 :param userdata_dir: userdata的存放路径''' options = None try: if kind == 'chrome': options = ChromeOptions() options.binary_location = driver.binary_location_chrome options.add_argument('--disable-gpu') #据说谷歌文档提到需要加上这个属性来规避bug elif kind == 'edge': options = EdgeOptions() options.use_chromium = True options.binary_location = driver.binary_location_edge elif kind == 'firefox': options = FirefoxOptions() options.binary_location = driver.binary_location_firefox else: raise Exception( 'error: only support chrome/firefox/edge, please check') if headless: options.add_argument('--headless') if disguise: options.add_experimental_option('excludeSwitches', ['enable-automation']) #firefox的options没有add_experimental_option if independent_userdata: if userdata_dir == '' or not os.path.isdir(userdata_dir): print( 'userdata_dir wrong, please input correct userdata_dir' ) else: userdata_dir = userdata_dir.replace( '\\', '\\\\') #此处仅为windows系统设置路径 options.add_argument( '--user-data-dir={}'.format(userdata_dir)) return options except: raise Exception('driver.get_option_obj failed') def get_default_options(self): '''生成浏览器的默认配置。''' userdata_dir = '' if self.kind == 'chrome': userdata_dir = driver.userdata_dir_chrome elif self.kind == 'edge': userdata_dir = driver.userdata_dir_edge elif self.kind == 'firefox': userdata_dir = driver.userdata_dir_firefox else: raise Exception( 'error: only support chrome/firefox/edge, please check') return driver.get_options_obj(self.kind, self.headless, self.disguise, self.independent_userdata, userdata_dir) def get_browser(self): '''根据配置返回浏览器。''' kind, driver_path = self.kind, driver.driver_path driver_browser, options = None, self.options try: if kind == 'chrome': driver_browser = Chrome(executable_path=os.path.join( driver_path, 'chromedriver.exe'), options=options) elif kind == 'edge': driver_browser = Edge(executable_path=os.path.join( driver_path, 'msedgedriver.exe'), options=options) elif kind == 'firefox': driver_browser = Firefox(executable_path=os.path.join( driver_path, 'geckodriver.exe'), options=options) return driver_browser except selenium.common.exceptions.TimeoutException: raise Exception('driver.get_browser failed') def get_page_source(self, url: str, XPath_expr: str = '', timeout: int = 60) -> str: '''通过浏览器对象获取目标网页源码。 :param url: 目标网页 :param XPath_expr: XPath表达式，用于让页面显式等待 :param timeout: 等待时间 :return 网页源码''' browser, log = self.browser, self.log """script = '''Object.defineProperty(navigator, 'webdriver', {get: ()= > undefined})''' browser.execute_cdp_cmd("Page.addScriptToEvaluateOnNewDocument", {"source": script}) #上面这段是运行js代码，将navigator的属性webdriver改为undefined，可隐藏爬虫身份 #如果设置了 disguise=True ，就不需要这个了""" try: browser.get(url) if XPath_expr != '': WebDriverWait(browser, timeout).until( EC.presence_of_element_located((By.XPATH, XPath_expr))) except selenium.common.exceptions.TimeoutException or selenium.common.exceptions.NoSuchElementException: browser.implicitly_wait(10) message = '浏览器等待超时，需要的元素仍未出现。隐式等待10s并返回页面源码，请检查是否正确' if log: log.info(message) else: print(message) return browser.page_source def get_tag_obj(self, url: str, XPath_expr: str, timeout: int = 60): '''通过浏览器对象获取目标标签。 :param url: 目标网页 :param XPath_expr: XPath表达式，一般来说就是目标标签，页面将为该标签显式等待 :param timeout: 等待时间 :return 标签对象''' browser, log = self.browser, self.log try: browser.get(url) element = WebDriverWait(browser, timeout).until( EC.presence_of_element_located((By.XPATH, XPath_expr))) return element except selenium.common.exceptions.TimeoutException or selenium.common.exceptions.NoSuchElementException: message = '浏览器等待超时，需要的元素仍未出现；或许XPath表达式书写错误，请检查' if log: log.info(message) else: print(message) return @staticmethod def help(): '''打印帮助信息。''' help_info = r"""注意，以下内容很多尚未验证。 目录：一、如何用 options 对象及其他选项设置浏览器的行为 二、browser对象的一些操作 三、对标签对象的鼠标操作 四、对标签对象的其他操作 五、显式等待与隐式等待 六、人工打开浏览器并交给程序控制 一、如何用 options 对象及其他选项设置浏览器的行为 1.options基本的设置方法（参数中用字符串指定设置）： （1）添加参数：options.add_argument() （2）添加扩展应用：options.add_extension() 和 options.add_encoded_extension() （3）添加实验性质的设置参数：options.add_experimental_option() 。firefox没有这个选项，所以仅用于chrome和edge 2.options一些常用的设置 （1）设置以开发者模式启动，并且webdriver相关的一些属性改回正常值。这个模式可以骗过爬虫识别 options.add_experimental_option('excludeSwitches', ['enable-automation']) （2）允许浏览器弹窗。如果需要用浏览器打开不只一个页面，必须把这个设置加上 options.add_argument("--disable-popup-blocking") （3）浏览器使用代理 options.add_argument("--proxy-server=http或https://代理ip:端口") （4）不加载图片。如果不爬取图片，这个选项可提升速度 options.add_argument('blink-settings=imagesEnabled=false') （5）禁用GPU显卡辅助渲染。谷歌文档提到需要加上这个属性来规避bug chrome_options.add_argument('--disable-gpu') （6）无头浏览器，即浏览器不提供可视化页面。无桌面的linux必须加这条 options.add_argument('--headless') （7）隐藏滚动条。可用于一些特殊页面 options.add_argument('--hide-scrollbars') （8）改变UA options.add_argument('user-agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36"') （9）指定浏览器分辨率 options.add_argument('window-size=1920x3000') （10）在沙箱之外运行（以最高权限运行） options.add_argument('--no-sandbox') （11）手动指定使用的浏览器位置 options.binary_location = r"浏览器可执行文件的路径" （12）添加插件（crx文件是chrome的插件文件） options.add_extension('crx文件路径') （13）禁用JavaScript options.add_argument("--disable-javascript") 二、browser对象的一些操作 在其他地方，browser对象也常被称为driver对象。 1.browser对象的元素定位 可以使用id、name、tag_name等定位。有如下方法可使用： browser.find_elements_by_id() browser.find_elements_by_name() browser.find_elements_by_class_name() browser.find_elements_by_tag_name() browser.find_elements_by_link_text() browser.find_elements_by_partial_link_text() browser.find_elements_by_xpath() browser.find_elements_by_css_selector() 其中最推荐的是最后两个，通过xpath和css_selector查找元素。 上述方法返回的都是匹配的标签序列；把 elements 改成 element 则只返回第一个匹配的标签。 另外，4.0之后的selenium据说不支持 find_elements_by_id 这些方法了，需要改成： from selenium.webdriver.common.by import By browser.find_element(By.ID, "ID名") browser.find_element(By.NAME, "标签名") 下文也有关于By的应用。 2.browser对象的其他方法和属性 （1）到前一页或后一页：browser.forward() 和 browser.back() （2）刷新：browser.refresh() （3）获取信息：browser有一些属性，比如 browser.page_source、browser.title、browser.current_url 分别为当前页面的源码、标题、网址 （4）切换窗口或框架： switch_to_window("windowName") 和 switch_to_frame("frameName") 。框架，比如弹出一个输入用户密码的小框框就叫框架。 （5）返回到父frame：switch_to_default_content() （6）切换到警告框：a = browser.switch_to_alert() 警告框可以像标签对象一样操作。方法和属性有：a.text、a.accept()、a.dismiss()、a.send_keys() ，分别是警告内容、接受警告、关闭警告、发送按键。send_keys方法详见后。 （7）操作cookie： 获得所有cookie信息（字典形式）：get_cookies() 返回cookie中键对应的值：get_cookie("键名") 添加键值对到cookie（对已存在的键可能会替换其值）：add_cookie(cookie_dict) 删除cookie信息：delete_cookie("键名", optionsString) 。其中 optionsString 是该cookie的选项，目前支持的选项包括 路径、域 删除所有cookie：delete_all_cookies() （8）执行javascript：execute_script(js_code) ，js_code是字符串形式的js代码 （9）屏幕截图：get_screenshot_as_file(r"图片保存路径") ，这个方法可用于无头时查找问题 （10）关闭与退出：close() 关闭单个窗口，quit() 关闭所有窗口并退出 （11）页面最多加载几秒：browser.set_page_load_timeout(timeout) 页面的js最多加载几秒：browser.set_script_timeout(timeout) 三、对标签对象的鼠标操作 鼠标操作称为动作链，在设置完所有动作后用perform()方法执行。 1.一般格式： from selenium.webdriver.common.action_chains import ActionChains #a为一个标签对象 a = browser.find_elements_by_xpath(r'XPath表达式') AC = ActionChains(browser).move_to_element(a) #其他动作 AC.perform() #设置完所有动作后用perform()方法执行 2.常用的鼠标操作 左键单击：click() 右键单击：context_click() 左键双击：double_click() 拖动并松开：drag_and_drop() 在元素处悬停：move_to_element() 四、对标签对象的其他操作 常见的有输入文本，提交，键盘按键。 （1）对一个标签输入文本。常用于填写用户名密码 browser.find_element_by_xpath(r'XPath表达式').send_keys("待输入的文本") （2）清除标签中的文本。输入框里如果有默认值，可以用这个方法清除 标签对象.clear() （3）提交表单：submit() （4）元素的尺寸：size （5）元素的内容：text （6）获得属性值：get_attribute("属性名") （7）元素是否可见：is_displayed() 键盘输入按键： BackSpace：标签对象.send_keys(Keys.BACK_SPACE) Space：send_keys(Keys.SPACE) Tab：send_keys(Keys.TAB) Esc：send_keys(Keys.ESCAPE) Enter：send_keys(Keys.ENTER) Ctrl+A：send_keys(Keys.CONTROL,'a') Ctrl+C：send_keys(Keys.CONTROL,'c') Ctrl+X：send_keys(Keys.CONTROL,'x') Ctrl+V：send_keys(Keys.CONTROL,'v') F1：send_keys(Keys.F1) 下拉框选择值： 定位到下拉框标签，然后用：Select(下拉框标签对象).select_by_value('50') 类似的方法选择值 五、显式等待与隐式等待 selenium一般是用来爬取一些有js的网页，而js加载需要一定时间 可以用显式等待和隐式等待来让浏览器对象度过这段时间，前者更优 隐式等待：等待固定的秒数，使用 browser.implicitly_wait(秒数) 即可 显式等待：等待某个元素被js加载出来，一般格式如下： from selenium.webdriver.common.by import By from selenium.webdriver.support.ui import WebDriverWait from selenium.webdriver.support import expected_conditions as EC WebDriverWait(browser, 5, 0.5).until(EC.presence_of_element_located((By.XPATH, "XPath表达式"))) WebDriverWait(浏览器对象，最多等待几秒，每过多少秒检查一次) ，超时后会抛出NoSuchElementException异常 WebDriverWait类除了until方法，还有until_not方法。 EC.presence_of_element_located() 方法需要传入 locator 作为参数，这里用的是 (By.XPATH, "XPath表达式") 六、人工打开浏览器并交给程序控制 有时想尽办法也会被网站识别出爬虫，这时可以试试人工打开浏览器并交给程序控制。以win10系统为例，流程如下： 1.在cmd或终端中输入命令、打开浏览器，比如： chrome.exe --remote-debugging-port=9222 --user-data-dir="D:\webdrivers\userdata_chrome" msedge.exe --remote-debugging-port=9222 --user-data-dir="D:\webdrivers\userdata_edge" firefox.exe --remote-debugging-port=9222 --user-data-dir="D:\webdrivers\userdata_firefox" --user-data-dir 参数用于设置临时的userdata文件夹，这个参数必须也加上。 如果无法找到.exe文件，可以在cmd中进入浏览器的根目录，或把根目录加到系统路径中。 2.在options对象中添加： options.add_experimental_option('debuggerAddress', '127.0.0.1:9222') 即可。 """ print(help_info)selenium的使用过程大概就是指定浏览器，指定浏览器设置（options），打开浏览器即可。详细的使用注意事项都在 driver().help() 里了，下面来试试自建库的功能。4.selenium个人库使用范例打开一个chrome浏览器，打开bing搜索：from MyPythonLib import spider #这一行是笔者的自建库 driver = spider.driver('chrome', headless=False) browser = driver.browser browser.get('http://bing.com') input()手动打开浏览器，交给selenium控制：以windows、chrome为例，在cmd中输入：cd C:\Program Files\Google\Chrome\Application chrome.exe --remote-debugging-port=9222 --user-data-dir="D:\webdrivers\userdata_chrome"注意 D:\webdrivers\userdata_chrome 需要提前创建好。最后在py文件中写入：from MyPythonLib import spider options = spider.driver.get_options_obj('chrome', headless=False, disguise=False) options.add_experimental_option('debuggerAddress', '127.0.0.1:9222') driver = spider.driver('chrome', options=options) browser = driver.browser browser.get('http://bing.com')就能看到自己刚才打开的浏览器从初始页面变成bing搜索。如果需要自己定制options，也按上面的方法，在构造driver时传入需要的options对象即可。笔者为了方便，添加了 spider.driver.get_options_obj() 方法，返回一个默认的options。
Python，爬虫与深度学习（6）——爬虫详解（二）构建自己的ip池（简易版）
步骤/目录： 1.需求分析 2.mysql建表 3.网页请求分析 4.爬虫编写 （0）安装所需模块 （1）mysql存取 （2）爬虫主体编写 （3）其他爬虫补充 （4）实际运行 （5）检测爬取的ip 5.总结与改进本文首发于个人博客https://lisper517.top/index.php/archives/44/，转载请注明出处。本文的目的是构建自己的ip池。本文写作日期为2022年9月4日。操作的平台为win10，编辑器为VS code。爬虫写的好，牢房进的早。使用爬虫时不要占用服务器太多流量，否则可能惹祸上身。除了python与爬虫，这里实际上还需要了解html格式与mysql命令。可以在 runoob-html 或 runoob-mysql 找到初步的学习资料，或者查看笔者其他的文章。最后，笔者的能力有限，也并非CS从业人员，很多地方难免有纰漏或者不符合代码原则的地方，请在评论中指出。1.需求分析之前说过，在对别人的网页服务器进行爬取时要注意不能占用太多服务器资源。但有时候，我们确实需要在短时间内对某个网页高频访问，又担心自己被封ip，那么此时维护一个ip池就非常有必要，我们需要对网络上的ip（每个ip背后是一个或一群服务器）进行测试，看是否能利用这些ip作为代理服务器访问其他网页。本文先介绍如何爬取ip，后续文章会介绍如何用自己的ip池爬取网页，以及代理失效时更换代理。在爬虫的一些较高级的应用中，一个ipool是非常基础且不可或缺的，所以建议学会scrapy后就应该尽快建立自己的ipool（除非没有高频访问某个网页的需求）。ip池的构建和使用分为两步，第一步是筛选ip。世界上的公网ipv4地址最多有 4,294,967,296 个（256^4。实际上有一些被局域网占用，有一些不会使用），要一个个去验证这些ip耗时太长，何况还需要对已存在的ip进行验证、剔除失效ip。所以，这里使用一些提供免费代理ip的网页缩小ip搜索范围，爬取到的代理ip不做验证，都放入库中（这里使用mysql）。第二步是对mysql中已有的ip进行验证，挑选出对某个网址有效的代理ip。这一步严格来讲和第一步关系不大，它可以在前一个步骤进行时也同步进行，更重要的是这一步是在其他爬虫开始后才进行，下面就讲解一下免费代理ip的使用。免费代理可分为HTTP或HTTPS，高匿名和低匿名。如果爬虫运行时经过了这些免费代理，毕竟还是别人的服务器，你的请求和响应是到了别人那里，所以用代理服务器时不要传递一些私密信息。实在有必要的话，自己买服务器做转发。最后，同一个代理ip，可能你用它代理一个网站可以，代理另一个网站就不行，这也是不确定因素，所以爬到的代理ip在使用时还要对特定的网址进行测试，其实也是比较麻烦的，如果有紧急需求，建议买一些付费的代理ip。2.mysql建表在mysql中存储收集到的ip，需要新建以下列：ip，字符串形式，VARCHAR(50)（为了将来兼容ipv6，稍微长一点）；端口号，数值形式，SMALLINT UNSIGNED（刚好是0~65535）；匿名度，2个bit位，BIT(2)（0表示低匿名，1表示高匿名，3表示未知）；类型（HTTP还是HTTPS），2个bit位，BIT(2)（0表示HTTP，1表示HTTPS，2表示都可，3表示其他）；ip物理位置，字符串，VARCHAR(100)；响应速度，MEDIUMINT，单位是毫秒ms；最后一次验证的时间，日期时间格式。建表的语句在程序中将给出。其实对这种不断更新、添加的数据，redis更加合适，有关redis的内容将在后续的文章中介绍。3.网页请求分析这里使用的提供代理ip的网站为（笔者给出的例子中，只使用了第一个网址）：http://www.ip3366.net/free/ https://www.kuaidaili.com https://ip.jiangxianli.com/ https://www.cool-proxy.net/ https://free-proxy-list.net https://www.ipaddress.com/proxy-list/ http://proxy-list.org/english/ https://www.proxynova.com/proxy-server-list/ http://pubproxy.com/api/proxy?limit=5&format=txt&type=http&level=anonymous&last_check=60&no_country=CN https://www.rmccurdy.com/.scripts/proxy/good.txt http://spys.one/en/anonymous-proxy-list/ https://raw.githubusercontent.com/sunny9577/proxy-scraper/master/proxies.json https://raw.githubusercontent.com/a2u/free-proxy-list/master/free-proxy-list.txt https://raw.githubusercontent.com/clarketm/proxy-list/master/proxy-list.txt https://raw.githubusercontent.com/TheSpeedX/SOCKS-List/master/http.txt这些网页的请求响应都挺简单的，这里就不专门介绍了。由于涉及多个网站，这里将使用多线程的方法，持续对各个ip网站爬取。唯一需要注意的是对每个网站，爬取ip的速度不要太快了，虽然对单个网站爬取速度不快，但毕竟有这么多ip网站，实际上ip入库的速度挺快的。4.爬虫编写（0）安装所需模块需要用到requests（处理网络请求）、bs4（对响应内容进行解析）、pymysql（连接mysql数据库）、lxml（网页解析器）、cryptography（连接mysql时密码加密）库，在cmd中使用 pip install requests bs4 pymysql lxml cryptography 下载（下载失败就一直使用上述命令，直到下载成功）。之后新建一个文件夹，用VS Code打开该文件夹、新建一个py文件，进行下面的操作。（1）mysql存取使用mysql存储ip，其他爬虫需要通过这个类存取ip。自己使用时，记得把mysql连接的地址、端口、密码更改一下。#!/usr/bin/python # -*- coding: UTF-8 -*- import pymysql, time, os, logging from traceback import print_exc if not os.path.isdir(os.path.join(os.getcwd(), 'logs')): os.mkdir(os.path.join(os.getcwd(), 'logs')) class MysqlOperationFailed(Exception): """若更改mysql出现错误则抛出这个异常。""" def __init__(self): message = "Failed to change data in MySQL, please check. " super().__init__(message) class mysql_api(): """通过这个界面实现提交爬取的ip、取出ip的操作。""" conn = pymysql.connect(host='127.0.0.1', port=53306, user='root', passwd='mysqlpasswd', db='mysql', charset='utf8') cur = conn.cursor() def __init__(self, log_name: str): """初始函数，需要提交一个log_name作为进程的log文件名。""" self.cur = mysql_api.conn.cursor() mysql_api.check_table() self.logging = logging.basicConfig( level=logging.INFO, filename=os.path.join( os.getcwd(), 'logs', '{}-log{}.txt'.format(log_name, time.strftime("%Y-%m-%d-%H_%M_%S")))) def __del__(self): self.cur.close() @staticmethod def check_table(): """检查一下mysql中是否已经建好了表，若表不存在则新建表。这里并没有检查：若表存在，表的格式是否正确。""" cur, conn = mysql_api.cur, mysql_api.conn command = ''' CREATE TABLE IF NOT EXISTS ipool ( no BIGINT UNSIGNED AUTO_INCREMENT, ip VARCHAR(50) NOT NULL DEFAULT "0.0.0.0" COMMENT "IP address", port SMALLINT UNSIGNED NOT NULL DEFAULT 0 COMMENT "port", anonymous BIT(2) NOT NULL DEFAULT 3 COMMENT "whether ip is anonymous", type BIT(2) NOT NULL DEFAULT 3 COMMENT "HTTP or HTTPS or both or others", physical_address VARCHAR(100) NOT NULL DEFAULT "unknown" COMMENT "where is the server", response_time MEDIUMINT NOT NULL DEFAULT -1 COMMENT "response_time in microseconds", last_verify DATETIME NOT NULL DEFAULT '1000-01-01 00:00:00' COMMENT "last verify time", created TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP, PRIMARY KEY (no));''' command = command.replace('\n', '') command = command.replace(' ', '') try: cur.execute("CREATE DATABASE IF NOT EXISTS ipool;") cur.execute("USE ipool;") cur.execute(command) conn.commit() except: conn.rollback() raise MysqlOperationFailed def save_ips(self, ips_list: list): """存入多个ip（以字典的列表形式提交）。注意这里只是对不存在的数据添加默认值，没有验证数据类型等。 另外注意，响应时间单位为ms。""" cur, conn = self.cur, mysql_api.conn command = '''INSERT INTO ipool (ip,port,anonymous,type,physical_address,response_time,last_verify) VALUES ''' for ip_dict in ips_list: ip, port, anonymous, type_, physical_address, response_time, last_verify = "0.0.0.0", 0, 3, 3, 'unknown', -1, "1000-01-01 00:00:00" if 'ip' in ip_dict.keys(): ip = ip_dict['ip'] if 'port' in ip_dict.keys(): port = ip_dict['port'] if 'anonymous' in ip_dict.keys(): anonymous = ip_dict['anonymous'] if 'type' in ip_dict.keys(): type_ = ip_dict['type'] if 'physical_address' in ip_dict.keys(): physical_address = ip_dict['physical_address'] if 'response_time' in ip_dict.keys(): response_time = ip_dict['response_time'] if 'last_verify' in ip_dict.keys(): last_verify = ip_dict['last_verify'] command += '("{}",{},{},{},"{}",{},"{}"),'.format( ip, port, anonymous, type_, physical_address, response_time, last_verify) command = command[:-1] + ';' try: cur.execute(command) conn.commit() except: conn.rollback() raise MysqlOperationFailed @staticmethod def check_repeat(): """检查爬到的数据里有没有重复的，只检查ip和port都相同的。如果有重复的，保留最新的一条记录，也就是no最大的。""" cur, conn = mysql_api.cur, mysql_api.conn cur.execute('USE ipool;') cur.execute( 'SELECT ip,port FROM ipool GROUP BY ip,port HAVING COUNT(*)>1;') repeated_items = cur.fetchall() if len(repeated_items) == 0: print('\n数据库中无重复数据。') return else: print('\n数据库中检测到重复数据，有{}种。'.format(len(repeated_items))) command = ''' DELETE FROM ipool WHERE no IN (SELECT no FROM (SELECT no FROM ipool WHERE (ip,port) IN (SELECT ip,port FROM ipool GROUP BY ip,port HAVING COUNT(*)>1) AND no NOT IN (SELECT MAX(no) FROM ipool GROUP BY ip,port HAVING COUNT(*)>1)) AS a);''' command = command.replace('\n', '') command = command.replace(' ', '') try: cur.execute(command) conn.commit() except: conn.rollback() raise MysqlOperationFailed @staticmethod def get_ips(anonymous: list = [1], type_: list = [2], response_time: int = 30000, last_verify_interval: str = '48:00:00', limit: int = 1000) -> tuple: """根据要求返回多个ip，只返回ip和port。 :param last_verify_interval 用于指定上次验证的时间距现在不超过多久。""" cur, conn = mysql_api.cur, mysql_api.conn cur.execute('USE ipool;') anonymous, type_ = tuple(anonymous), tuple(type_) command = ''' SELECT ip,port FROM ipool WHERE anonymous IN {} AND type IN {} AND response_time BETWEEN 0 AND {} AND (NOW()-last_verify)<"{}" ORDER BY response_time LIMIT {}; '''.format(anonymous, type_, response_time, last_verify_interval, limit) command = command.replace('\n', '') command = command.replace(' ', '') print(command) cur.execute(command) return cur.fetchall() if __name__ == "__main__": try: a = mysql_api('kuaidaili') ip_list = [] for i in range(1, 256): ip_dict = { 'ip': '192.168.1.{}'.format(i), 'port': i, 'response_time': i * 10, 'last_verify': '2022-09-05 12:00:00' } ip_list.append(ip_dict) a.save_ips(ip_list) a.check_repeat() print(a.get_ips([0, 1, 2, 3], [0, 1, 2, 3], 30000)) except: print_exc()（2）爬虫主体编写这里稍微把前面mysql存取的部分进行了更改，一并写出。单个ip网站的爬虫使用了递归调用的方法，不太适合页码比较多的情况。这里只是写了对 http://www.ip3366.net/free/ 的爬虫、并用线程的方法开启它，其他爬虫可在后面自行补充。最后，自己使用时，记得把mysql连接的地址、端口、密码更改一下。#!/usr/bin/python # -*- coding: UTF-8 -*- import pymysql, os, time, logging from traceback import print_exc if not os.path.isdir(os.path.join(os.getcwd(), 'logs')): os.mkdir(os.path.join(os.getcwd(), 'logs')) logging.basicConfig(level=logging.INFO, filename=os.path.join( os.getcwd(), 'logs', 'log-{}.txt'.format( time.strftime("%Y-%m-%d-%H_%M_%S")))) class MysqlOperationFailed(Exception): """若更改mysql出现错误则抛出这个异常。""" def __init__(self): message = "Failed to change data in MySQL, please check. " super().__init__(message) class mysql_api(): """通过这个界面实现提交爬取的ip、取出ip的操作。""" conn = pymysql.connect(host='127.0.0.1', port=53306, user='root', passwd='mysqlpasswd', db='mysql', charset='utf8') cur = conn.cursor() def __init__(self): self.cur = mysql_api.conn.cursor() mysql_api.check_table() message = 'one of mysql_api object started' logging.info(message) def __del__(self): self.cur.close() message = 'one of mysql_api object stopped' logging.info(message) @staticmethod def check_table(): """检查一下mysql中是否已经建好了表，若表不存在则新建表。这里并没有检查：若表存在，表的格式是否正确。""" cur, conn = mysql_api.cur, mysql_api.conn command = ''' CREATE TABLE IF NOT EXISTS ipool ( no BIGINT UNSIGNED AUTO_INCREMENT, ip VARCHAR(50) NOT NULL DEFAULT "0.0.0.0" COMMENT "IP address", port SMALLINT UNSIGNED NOT NULL DEFAULT 0 COMMENT "port", anonymous BIT(2) NOT NULL DEFAULT 3 COMMENT "whether ip is anonymous", type BIT(2) NOT NULL DEFAULT 3 COMMENT "HTTP or HTTPS or both or others", physical_address VARCHAR(100) NOT NULL DEFAULT "unknown" COMMENT "where is the server", response_time MEDIUMINT NOT NULL DEFAULT -1 COMMENT "response_time in microseconds", last_verify DATETIME NOT NULL DEFAULT '1000-01-01 00:00:00' COMMENT "last verify time", created TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP, PRIMARY KEY (no));''' command = command.replace('\n', '') command = command.replace(' ', '') try: cur.execute("CREATE DATABASE IF NOT EXISTS ipool;") cur.execute("USE ipool;") cur.execute(command) conn.commit() message = 'database ipool and table ipool checked' logging.info(message) except: conn.rollback() raise MysqlOperationFailed def save_ips(self, ips_list: list): """存入多个ip（以字典的列表形式提交）。注意这里只是对不存在的数据添加默认值，没有验证数据类型等。 另外注意，响应时间单位为ms。""" cur, conn = self.cur, mysql_api.conn command = '''INSERT INTO ipool (ip,port,anonymous,type,physical_address,response_time,last_verify) VALUES ''' for ip_dict in ips_list: ip, port, anonymous, type_, physical_address, response_time, last_verify = "0.0.0.0", 0, 3, 3, 'unknown', -1, "1000-01-01 00:00:00" if 'ip' in ip_dict.keys(): ip = ip_dict['ip'] if 'port' in ip_dict.keys(): port = ip_dict['port'] if 'anonymous' in ip_dict.keys(): anonymous = ip_dict['anonymous'] if 'type' in ip_dict.keys(): type_ = ip_dict['type'] if 'physical_address' in ip_dict.keys(): physical_address = ip_dict['physical_address'] if 'response_time' in ip_dict.keys(): response_time = ip_dict['response_time'] if 'last_verify' in ip_dict.keys(): last_verify = ip_dict['last_verify'] command += '("{}",{},{},{},"{}",{},"{}"),'.format( ip, port, anonymous, type_, physical_address, response_time, last_verify) message = 'trying to save ip:port={}:{}'.format(ip, port) logging.info(message) command = command[:-1] + ';' try: cur.execute(command) conn.commit() message = 'saving ips successfully' logging.info(message) except: conn.rollback() raise MysqlOperationFailed def check_repeat(self): """检查爬到的数据里有没有重复的，只检查ip和port都相同的。如果有重复的，保留最新的一条记录，也就是no最大的。""" cur, conn = mysql_api.cur, mysql_api.conn cur.execute('USE ipool;') cur.execute( 'SELECT ip,port FROM ipool GROUP BY ip,port HAVING COUNT(*)>1;') repeated_items = cur.fetchall() if len(repeated_items) == 0: message = 'no repeated ip:port in ipool' logging.info(message) return else: message = 'found ip:port repeated {} kinds'.format( len(repeated_items)) logging.info(message) command = ''' DELETE FROM ipool WHERE no IN (SELECT no FROM (SELECT no FROM ipool WHERE (ip,port) IN (SELECT ip,port FROM ipool GROUP BY ip,port HAVING COUNT(*)>1) AND no NOT IN (SELECT MAX(no) FROM ipool GROUP BY ip,port HAVING COUNT(*)>1)) AS a);''' command = command.replace('\n', '') command = command.replace(' ', '') try: cur.execute(command) conn.commit() message = 'repeated ip:port deleted successfully. ' logging.info(message) except: conn.rollback() raise MysqlOperationFailed @staticmethod def get_ips(anonymous: list = [1], type_: list = [2], response_time: int = 30000, last_verify_interval: str = '48:00:00', limit: int = 1000) -> tuple: """根据要求返回多个ip，只返回ip和port。 :param last_verify_interval 用于指定上次验证的时间距现在不超过多久。""" cur, conn = mysql_api.cur, mysql_api.conn cur.execute('USE ipool;') anonymous, type_ = tuple(anonymous), tuple(type_) command = ''' SELECT ip,port FROM ipool WHERE anonymous IN {} AND type IN {} AND response_time BETWEEN 0 AND {} AND (NOW()-last_verify)<"{}" ORDER BY response_time LIMIT {}; '''.format(anonymous, type_, response_time, last_verify_interval, limit) command = command.replace('\n', '') command = command.replace(' ', '') print(command) cur.execute(command) ips = cur.fetchall() return ips import threading, requests, re from lxml import etree headers = { 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36' } class spider_threads(): """爬虫类，通过多线程的方法同时爬取多个网站。""" websites = [ 'http://www.ip3366.net/free/' ] url_spidered_list = [] for i in range(len(websites)): url_spidered_list.append(set()) def __init__(self): self.threads = len(spider_threads.websites) @staticmethod def get_session(): """返回一个session对象。""" session = requests.Session() session.mount('http://', requests.adapters.HTTPAdapter(max_retries=3)) session.mount('https://', requests.adapters.HTTPAdapter(max_retries=3)) return session class spider_website(): """定义一个网站爬虫的基本操作。""" def __init__(self, no: int): self.session = spider_threads.get_session() self.base_url = spider_threads.websites[no] self.url_spidered = spider_threads.url_spidered_list[no] self.kind = re.search(r'\.{1}.+\.{1}', self.base_url).group().replace('.', '') self.mysql_api = mysql_api() def get_html(self, url): """对一个页面，返回其响应对象。""" global headers session = self.session try: html = session.get(url, timeout=30, headers=headers) except requests.exceptions.TooManyRedirects as e: message = "网页重定向，休息10min（{}）。".format(self.kind) logging.warning(message) print('\r' + message, end='') time.sleep(600) html = session.get(url, timeout=30, headers=headers) html.encoding = html.apparent_encoding return html def get_ip_recursively(self, url): """这个方法用于递归调用爬虫，将在子类中重写。""" pass def start_spider(self): """这个方法用于开启爬虫，将在子类中重写。""" pass class spider_ip3366(spider_website): """ip3366的爬虫类。""" def __init__(self): super().__init__(0) self.start_urls = [ self.base_url, 'http://www.ip3366.net/free/?stype=2' ] def get_ip_recursively(self, url): """这是一个递归调用的爬虫。它将爬取页面上的信息，将爬取过的网页存入字典，对页面上新的网页进行爬取。 重写了父类方法。""" html = self.get_html(url) time.sleep(15) message = '1 page crawled:{}'.format(url) logging.info(message) html = etree.HTML(html.content) ip_nodes = html.xpath(r'//*[@id="list"]/table/tbody/tr') ips_list = [] for ip_node in ip_nodes: ip_dict = {} ip = ip_node.xpath(r'./td[1]/text()')[0].strip() port = int(ip_node.xpath(r'./td[2]/text()')[0].strip()) anonymous = (0, 1)['高匿' in ip_node.xpath(r'./td[3]/text()')[0]] type_ = (0, 1)['HTTPS' in ip_node.xpath(r'./td[4]/text()')[0]] physical_address = ip_node.xpath(r'./td[5]/text()')[0].strip() response_time = 1000 * int( ip_node.xpath(r'./td[6]/text()')[0].strip().replace( '秒', '')) response_time = (response_time, 250)[response_time == 0] last_verify = ip_node.xpath( r'./td[7]/text()')[0].strip().replace('/', '-') ip_dict['ip'] = ip ip_dict['port'] = port ip_dict['anonymous'] = anonymous ip_dict['type'] = type_ ip_dict['physical_address'] = physical_address ip_dict['response_time'] = response_time ip_dict['last_verify'] = last_verify ips_list.append(ip_dict) self.mysql_api.save_ips(ips_list) self.url_spidered.add(url) page_urls = html.xpath(r'//*[@id="listnav"]/ul/a/@href') for i in range(len(page_urls)): page_url = self.base_url + page_urls[i] if page_url not in self.url_spidered: self.get_ip_recursively(page_url) else: return def start_spider(self): """这个方法用于开启爬虫，对父类的同名方法进行重写。""" for start_url in self.start_urls: self.get_ip_recursively(start_url) self.mysql_api.check_repeat() def start_all_spider_process(): process_list = [] process_list.append(crawl_website_ip3366().start_spider) for i in process_list: process_pool.apply_async(i) time.sleep(35) process_pool.close() process_pool.join() if __name__ == "__main__": try: start_all_spider_process() except: traceback.print_exc()spider_ip3366类是一个范例，其他爬虫也照着这样写就行了，所以剩下的只是简单的重复劳动，这里笔者就不进行展示了。如果要继续写的话，注意其他网站的爬虫都和spider_ip3366类处于同一个层级，也在spider_threads内部。添加其他网站的爬虫后，在start_all_spider_threads里添加线程即可。（4）实际运行自行添加其他网站的爬虫后，运行。（5）检测爬取的ip这是相对独立的一个步骤，并且要在使用时同步进行，将在后续的文章中讲解。5.总结与改进（1）这个爬虫使用了递归，在内存小一点的机器上其实不太好运行。有可能的话，可以改用递归外的方法（比如用scrapy），专门买一个更加小巧的机器或云服务器，全天运行这个爬虫和ip检验，以便能及时更新自己的ipool。（2）在上一个例子（ Python，爬虫与深度学习（2）——爬虫详解（一）爬取金融机构成交持仓排名 ）和本例中，都用到了一些重复的方法，比如连接mysql时添加conn、cur，以及log操作。这部分其实可以在自建库里写方法调用，也免得每次都要更改。（3）这里所有爬虫使用的log文件都是同一个，不太容易看清楚。后续可以让每个爬虫使用不同的log文件。（4）除了自己写这种ipool爬虫，其实在github上有其他人做好的成品。笔者在这里推荐两个项目： https://github.com/SpiderClub/haipproxy 和 https://github.com/imWildCat/scylla ，这两个项目都可以用docker安装。更新：docker-scylla可能自带挖矿病毒，详见这个问题： https://github.com/imWildCat/scylla/issues/102 。