通過腳本同時運行幾個spider

# 通過腳本同時運行幾個spider
目錄結構：

 

1.在命令行能通過的情況下創建兩個spider如
TestSpider
Test2Spider

2.在items.py的同級目錄創建run.py文件，有三種方式，任選其一，其代碼如下：

方式一： 通過CrawlerProcess同時運行幾個spider

run_by_CrawlerProcess.py源代碼：

# 通過CrawlerProcess同時運行幾個spider
from scrapy.crawler import CrawlerProcess
# 導入獲取項目配置的模塊
from scrapy.utils.project import get_project_settings
# 導入蜘蛛模塊(即自己創建的spider)
from spiders.test import TestSpider
from spiders.test2 import Test2Spider

# get_project_settings() 必須得有，不然"HTTP status code is not handled or not allowed"
process = CrawlerProcess(get_project_settings())
process.crawl(TestSpider) # 注意引入
#process.crawl(Test2Spider) # 注意引入
process.start()

方式二：通過CrawlerRunner同時運行幾個spider

 run_by_CrawlerRunner.py源代碼：

# 通過CrawlerRunner同時運行幾個spider
from twisted.internet import reactor
from scrapy.crawler import CrawlerRunner
from scrapy.utils.log import configure_logging
# 導入獲取項目配置的模塊
from scrapy.utils.project import get_project_settings
# 導入蜘蛛模塊(即自己創建的spider)
from spiders.test import TestSpider
from spiders.test2 import Test2Spider

configure_logging()
# get_project_settings() 必須得有，不然"HTTP status code is not handled or not allowed"
runner = CrawlerRunner(get_project_settings())
runner.crawl(TestSpider)
#runner.crawl(Test2Spider)
d = runner.join()
d.addBoth(lambda _: reactor.stop())
reactor.run() # the script will block here until all crawling jobs are finished

方式三：通過CrawlerRunner和鏈接(chaining) deferred來線性運行來同時運行幾個spider

run_by_CrawlerRunner_and_Deferred.py源代碼：

# 通過CrawlerRunner和鏈接(chaining) deferred來線性運行來同時運行幾個spider
from twisted.internet import reactor, defer
from scrapy.crawler import CrawlerRunner
from scrapy.utils.log import configure_logging
# 導入獲取項目配置的模塊
from scrapy.utils.project import get_project_settings
# 導入蜘蛛模塊(即自己創建的spider)
from spiders.test import TestSpider
from spiders.test2 import Test2Spider

configure_logging()
# get_project_settings() 必須得有，不然"HTTP status code is not handled or not allowed"
runner = CrawlerRunner(get_project_settings())

@defer.inlineCallbacks
def crawl():
    yield runner.crawl(TestSpider) 
    #yield runner.crawl(Test2Spider) 
    reactor.stop()

crawl()
reactor.run() # the script will block here until the last crawl call is finished

 

3.修改兩個spider文件引入items,和外部類的如(HeadersHelper.py)的引入模式（以run.py所在目錄為中心）
原導入模式：

from ..items import ScrapydoubanmovieItem
from .HeadersHelper import HeadersHelper

注釋：這種導入能夠在命令行scrapy crawl Test正常運行

修改為：

from items import ScrapydoubanmovieItem
from .HeadersHelper import HeadersHelper

注釋：修改后這種導入在命令行scrapy crawl Test會報錯，但通過運行run.py文件，能夠同時運行兩個spider

4.按照運行python文件的方式運行run.py,可以得到結果