通常開發好的Scrapy爬蟲部署到服務器上，要不使用nohup命令，要不使用scrapyd。如果使用nohup命令的話，爬蟲掛掉了，你可能還不知道，你還得上服務器上查或者做額外的郵件通知操作。如果使用scrapyd，就是部署爬蟲的時候有點復雜，功能少了點，其他還好。
SpiderKeeper是一款管理爬蟲的軟件，和scrapinghub的部署功能差不多，能多台服務器部署爬蟲，定時執行爬蟲，查看爬蟲日志，查看爬蟲執行情況等功能。
項目地址：https://github.com/DormyMo/SpiderKeeper
一、運行環境
- Centos7
- Python2.7
- Python3.6
注意：supervisor用的是Python2.7，scrapyd用的是Python3.6，需要自行編譯安裝。Python3具體安裝自行百度。
二、安裝依賴
1、supervisor pip install supervisor
2、scrapyd pip3 install scrapyd
3、SpiderKeeperpip3 install SpiderKeeper
三、配置scrapyd
1、新建scrapyd的配置文件：
[scrapyd]
eggs_dir = eggs
logs_dir = logs
items_dir =
jobs_to_keep = 5
dbs_dir = dbs
max_proc = 0
max_proc_per_cpu = 4
finished_to_keep = 100
poll_interval = 5.0
bind_address = 0.0.0.0
http_port = 6800
debug = off
runner = scrapyd.runner
application = scrapyd.app.application
launcher = scrapyd.launcher.Launcher
webroot = scrapyd.website.Root
[services]
schedule.json = scrapyd.webservice.Schedule
cancel.json = scrapyd.webservice.Cancel
addversion.json = scrapyd.webservice.AddVersion
listprojects.json = scrapyd.webservice.ListProjects
listversions.json = scrapyd.webservice.ListVersions
listspiders.json = scrapyd.webservice.ListSpiders
delproject.json = scrapyd.webservice.DeleteProject
delversion.json = scrapyd.webservice.DeleteVersion
listjobs.json = scrapyd.webservice.ListJobs
daemonstatus.json = scrapyd.webservice.DaemonStatus
四、配置supervisor
1、創建配置的文件夾和配置文件
mkdir /etc/supervisor
echo_supervisord_conf > /etc/supervisor/supervisord.conf
2、編輯配置文件vim /etc/supervisor/supervisord.conf
;[include]
;files = relative/directory/*.ini
改為
[include]
files = conf.d/*.conf
3、新建conf.d文件夾mkdir /etc/supervisor/conf.d
4、添加scrapyd的配置文件vim /etc/supervisor/conf.d/scrapyd.conf
[program:scrapyd]
command=/usr/local/python3.5/bin/scrapyd
directory=/opt/SpiderKeeper
user=root
stderr_logfile=/var/log/scrapyd.err.log
stdout_logfile=/var/log/scrapyd.out.log
5、添加spiderkeeper的配置文件vim /etc/supervisor/conf.d/spiderkeeper.conf
[program:spiderkeeper]
command=spiderkeeper --server=http://localhost:6800
directory=/opt/SpiderKeeper
user=root
stderr_logfile=/var/log/spiderkeeper.err.log
stdout_logfile=/var/log/spiderkeeper.out.log
6、啟動supervisor，supervisord -c /etc/supervisor/supervisord.conf
五、使用
1、登錄http://localhost:5000
2、新建project
3、打包爬蟲文件
pip3 install scrapyd-client
scrapyd-deploy --build-egg output.egg
4、上傳打包好的爬蟲egg文件
SpiderKeeper可以識別多台服務器的scrapyd，具體多加--server就好。