在linux下安装并运行scrapyd


系统:centos7.4

安装scrapyd:pip isntall scrapyd

因为我腾讯云上是python2与python3并存的 所以我执行的命令是:pip3 isntall scrapyd

安装后新建一个配置文件:

  sudo mkdir /etc/scrapyd

  sudo vim /etc/scrapyd/scrapyd.conf

写入如下内容:(给内容在https://scrapyd.readthedocs.io/en/stable/config.html可找到)

[scrapyd] eggs_dir = eggs logs_dir = logs items_dir = jobs_to_keep = 5 dbs_dir = dbs max_proc = 0 max_proc_per_cpu = 10 finished_to_keep = 100 poll_interval = 5.0 bind_address = 0.0.0.0 http_port = 6800 debug = off runner = scrapyd.runner application = scrapyd.app.application launcher = scrapyd.launcher.Launcher webroot = scrapyd.website.Root [services] schedule.json = scrapyd.webservice.Schedule cancel.json = scrapyd.webservice.Cancel addversion.json = scrapyd.webservice.AddVersion listprojects.json = scrapyd.webservice.ListProjects listversions.json = scrapyd.webservice.ListVersions listspiders.json = scrapyd.webservice.ListSpiders delproject.json = scrapyd.webservice.DeleteProject delversion.json = scrapyd.webservice.DeleteVersion listjobs.json = scrapyd.webservice.ListJobs daemonstatus.json = scrapyd.webservice.DaemonStatus

主要更改bind_address=0.0.0.0

创建文件后执行命令启动scrapyd:   (scrapyd > /dev/null &)  当想要记录输出日志时: (scrapyd > /root/scrapyd.log &)

坑1:当我执行完命令后报错,说是找不到命令:

那是因为我系统上python2与3并存,所以找不到,这时应该做软连接:

我的python3路径:  /usr/local/python3

制作软连接: ln -s /usr/local/python3/bin/scrapy  /usr/bin/scrapy

昨晚软连接后,执行上边命令,又报错:

坑2:

这个好像是那个配置文件的最后一行有问题,具体原因不大清楚,我将最后一行删除,再次重新执行,scrapyd就跑起来了

 

 

想了解更多Python关于爬虫、数据分析的内容,欢迎大家关注我的微信公众号:悟道Python

  


免责声明!

本站转载的文章为个人学习借鉴使用,本站对版权不负任何法律责任。如果侵犯了您的隐私权益,请联系本站邮箱yoyou2525@163.com删除。



 
粤ICP备18138465号  © 2018-2025 CODEPRJ.COM