爬蟲的自我修養_5
一、CrawlSpiders類簡介
通過下面的命令可以快速創建 CrawlSpider模板 的代碼:
scrapy genspider -t crawl tencent tencent.com
上一個案例中,我們通過正則表達式,制作了新的url作為Request請求參數,現在我們可以換個花樣...
class scrapy.spiders.CrawlSpider
它是Spider的派生類,Spider類的設計原則是只爬取start_url列表中的網頁,而CrawlSpider類定義了一些規則(rule)來提供跟進link的方便的機制,從爬取的網頁中獲取link並繼續爬取的工作更適合。
源碼參考
class CrawlSpider(Spider):
rules = ()
def __init__(self, *a, **kw):
super(CrawlSpider, self).__init__(*a, **kw)
self._compile_rules()
#首先調用parse()來處理start_urls中返回的response對象
#parse()則將這些response對象傳遞給了_parse_response()函數處理,並設置回調函數為parse_start_url()
#設置了跟進標志位True
#parse將返回item和跟進了的Request對象
def parse(self, response):
return self._parse_response(response, self.parse_start_url, cb_kwargs={}, follow=True)
#處理start_url中返回的response,需要重寫
def parse_start_url(self, response):
return []
def process_results(self, response, results):
return results
#從response中抽取符合任一用戶定義'規則'的鏈接,並構造成Resquest對象返回
def _requests_to_follow(self, response):
if not isinstance(response, HtmlResponse):
return
seen = set()
#抽取之內的所有鏈接,只要通過任意一個'規則',即表示合法
for n, rule in enumerate(self._rules):
links = [l for l in rule.link_extractor.extract_links(response) if l not in seen]
#使用用戶指定的process_links處理每個連接
if links and rule.process_links:
links = rule.process_links(links)
#將鏈接加入seen集合,為每個鏈接生成Request對象,並設置回調函數為_repsonse_downloaded()
for link in links:
seen.add(link)
#構造Request對象,並將Rule規則中定義的回調函數作為這個Request對象的回調函數
r = Request(url=link.url, callback=self._response_downloaded)
r.meta.update(rule=n, link_text=link.text)
#對每個Request調用process_request()函數。該函數默認為indentify,即不做任何處理,直接返回該Request.
yield rule.process_request(r)
#處理通過rule提取出的連接,並返回item以及request
def _response_downloaded(self, response):
rule = self._rules[response.meta['rule']]
return self._parse_response(response, rule.callback, rule.cb_kwargs, rule.follow)
#解析response對象,會用callback解析處理他,並返回request或Item對象
def _parse_response(self, response, callback, cb_kwargs, follow=True):
#首先判斷是否設置了回調函數。(該回調函數可能是rule中的解析函數,也可能是 parse_start_url函數)
#如果設置了回調函數(parse_start_url()),那么首先用parse_start_url()處理response對象,
#然后再交給process_results處理。返回cb_res的一個列表
if callback:
#如果是parse調用的,則會解析成Request對象
#如果是rule callback,則會解析成Item
cb_res = callback(response, **cb_kwargs) or ()
cb_res = self.process_results(response, cb_res)
for requests_or_item in iterate_spider_output(cb_res):
yield requests_or_item
#如果需要跟進,那么使用定義的Rule規則提取並返回這些Request對象
if follow and self._follow_links:
#返回每個Request對象
for request_or_item in self._requests_to_follow(response):
yield request_or_item
def _compile_rules(self):
def get_method(method):
if callable(method):
return method
elif isinstance(method, basestring):
return getattr(self, method, None)
self._rules = [copy.copy(r) for r in self.rules]
for rule in self._rules:
rule.callback = get_method(rule.callback)
rule.process_links = get_method(rule.process_links)
rule.process_request = get_method(rule.process_request)
def set_crawler(self, crawler):
super(CrawlSpider, self).set_crawler(crawler)
self._follow_links = crawler.settings.getbool('CRAWLSPIDER_FOLLOW_LINKS', True)
二、LinkExtractors
Link Extractors 的目的很簡單: 提取鏈接。
每個LinkExtractor有唯一的公共方法是 extract_links(),它接收一個 Response 對象,並返回一個 scrapy.link.Link 對象。
Link Extractors要實例化一次,並且 extract_links 方法會根據不同的 response 調用多次提取鏈接。
主要參數
class scrapy.linkextractors.LinkExtractor(
allow = (), # 滿足括號中“正則表達式”的值會被提取,如果為空,則全部匹配
deny = (), # 與這個正則表達式(或正則表達式列表)不匹配的URL一定不提取
allow_domains = (), # 會被提取的鏈接的domains
deny_domains = (), # 一定不會被提取鏈接的domains
deny_extensions = None,
restrict_xpaths = (), # 使用xpath表達式,和allow共同作用過濾鏈接(一般只用allow就行了)
tags = ('a','area'),
attrs = ('href'),
canonicalize = True,
unique = True,
process_value = None
)
三、LinkExtractors
在rules中包含一個或多個Rule對象,每個Rule對爬取網站的動作定義了特定操作。如果多個rule匹配了相同的鏈接,則根據規則在本集合中被定義的順序,第一個會被使用。
主要參數
class scrapy.spiders.Rule(
link_extractor,
callback = None,
cb_kwargs = None,
follow = None,
process_links = None,
process_request = None
)
-
link_extractor:是一個Link Extractor對象,用於定義需要提取的鏈接。 -
callback: 從link_extractor中每獲取到鏈接時,參數所指定的值作為回調函數,該回調函數接受一個response作為其第一個參數。注意:當編寫爬蟲規則時,避免使用parse作為回調函數。由於CrawlSpider使用parse方法來實現其邏輯,如果覆蓋了 parse方法,crawl spider將會運行失敗。
-
follow:是一個布爾(boolean)值,指定了根據該規則從response提取的鏈接是否需要跟進。 如果callback為None,follow 默認設置為True ,否則默認為False。 -
process_links:指定該spider中哪個的函數將會被調用,從link_extractor中獲取到鏈接列表時將會調用該函數。該方法主要用來過濾。 -
process_request:指定該spider中哪個的函數將會被調用, 該規則提取到每個request時都會調用該函數。 (用來過濾request)
小Tips
由於CrawlSpider使用parse方法來實現其邏輯,如果覆蓋了 parse方法,crawl spider將會運行失敗。
四、Logging
Scrapy提供了log功能,可以通過 logging 模塊使用。
可以修改配置文件settings.py,任意位置添加下面兩行。
LOG_FILE = "TencentSpider.log"
LOG_LEVEL = "INFO"
Log levels
-
Scrapy提供5層logging級別:
-
CRITICAL - 嚴重錯誤(critical)
- ERROR - 一般錯誤(regular errors)
- WARNING - 警告信息(warning messages)
- INFO - 一般信息(informational messages)
- DEBUG - 調試信息(debugging messages)
logging設置
通過在setting.py中進行以下設置可以被用來配置logging:
LOG_ENABLED默認: True,啟用loggingLOG_ENCODING默認: 'utf-8',logging使用的編碼LOG_FILE默認: None,在當前目錄里創建logging輸出文件的文件名LOG_LEVEL默認: 'DEBUG',log的最低級別LOG_STDOUT默認: False 如果為 True,進程所有的標准輸出(及錯誤)將會被重定向到log中。例如,執行 print "hello" ,其將會在Scrapy log中顯示。
示例1、使用CrawlSpider爬取騰訊招聘網站
爬蟲模塊
# -*- coding: utf-8 -*-
import scrapy
from scrapy.linkextractors import LinkExtractor # 導入鏈接規則匹配類,用來提取符合規則的連接
from scrapy.spiders import CrawlSpider, Rule # 導入CrawlSpider類和Rule
from day_31.TencentCrawlSpider.TencentCrawlSpider.items import TencentcrawlspiderItem
class TencentSpider(CrawlSpider):
name = 'tencent'
allowed_domains = ['tencent.com']
start_urls = ['http://hr.tencent.com/position.php?&start=0']
rules = (
Rule(LinkExtractor(allow=r'position\.php\?&start=\d+#a'), callback='parse_item', follow=True),
# Response里鏈接的提取規則,返回的符合匹配規則的鏈接匹配對象的列表
# 獲取這個列表里的鏈接,依次發送請求,並且繼續跟進,調用指定回調函數處理
# 前面加r表示將正則表達式編譯成一個規則的對象
)
# 指定的回調函數
def parse_item(self, response):
for i in response.xpath('//tr[@class="even"] | //tr[@class="odd"]'):
item = TencentcrawlspiderItem()
item['name'] = i.xpath(".//a/text()").extract()[0]
item['link'] = i.xpath(".//a/@href").extract()[0]
item['type'] = i.xpath("./td[2]/text()").extract()[0]
item['number'] = i.xpath(".//td[3]/text()").extract()[0]
item['place'] = i.xpath(".//td[4]/text()").extract()[0]
item['rtime'] = i.xpath(".//td[5]/text()").extract()[0]
yield item
管道模塊
# -*- coding: utf-8 -*-
# Define your item pipelines here
#
# Don't forget to add your pipeline to the ITEM_PIPELINES setting
# See: http://doc.scrapy.org/en/latest/topics/item-pipeline.html
import json
class TencentcrawlspiderPipeline(object):
def __init__(self):
self.file = open('tencent-job.json','wb')
def process_item(self, item, spider):
text = json.dumps(dict(item),ensure_ascii=False)+'\n'
self.file.write(text.encode('utf-8'))
return item
def close_spider(self, spider):
self.file.close()
1 import scrapy 2 3 class TencentcrawlspiderItem(scrapy.Item): 4 # define the fields for your item here like: 5 name = scrapy.Field() 6 link = scrapy.Field() 7 type = scrapy.Field() 8 number = scrapy.Field() 9 place = scrapy.Field() 10 rtime = scrapy.Field()
1 # -*- coding: utf-8 -*- 2 3 # Scrapy settings for TencentCrawlSpider project 4 # 5 # For simplicity, this file contains only settings considered important or 6 # commonly used. You can find more settings consulting the documentation: 7 # 8 # http://doc.scrapy.org/en/latest/topics/settings.html 9 # http://scrapy.readthedocs.org/en/latest/topics/downloader-middleware.html 10 # http://scrapy.readthedocs.org/en/latest/topics/spider-middleware.html 11 12 BOT_NAME = 'TencentCrawlSpider' 13 14 SPIDER_MODULES = ['TencentCrawlSpider.spiders'] 15 NEWSPIDER_MODULE = 'TencentCrawlSpider.spiders' 16 17 18 # Crawl responsibly by identifying yourself (and your website) on the user-agent 19 #USER_AGENT = 'TencentCrawlSpider (+http://www.yourdomain.com)' 20 21 # Obey robots.txt rules 22 # ROBOTSTXT_OBEY = True 23 24 # Configure maximum concurrent requests performed by Scrapy (default: 16) 25 #CONCURRENT_REQUESTS = 32 26 27 # Configure a delay for requests for the same website (default: 0) 28 # See http://scrapy.readthedocs.org/en/latest/topics/settings.html#download-delay 29 # See also autothrottle settings and docs 30 DOWNLOAD_DELAY = 3 31 # The download delay setting will honor only one of: 32 #CONCURRENT_REQUESTS_PER_DOMAIN = 16 33 #CONCURRENT_REQUESTS_PER_IP = 16 34 35 # Disable cookies (enabled by default) 36 #COOKIES_ENABLED = False 37 38 # Disable Telnet Console (enabled by default) 39 #TELNETCONSOLE_ENABLED = False 40 41 # Override the default request headers: 42 DEFAULT_REQUEST_HEADERS = { 43 'User-Agent':'Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0;', 44 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', 45 # 'Accept-Language': 'en', 46 } 47 48 LOG_FILE = 'tencentlog.txt' 49 LOG_LEVEL = 'DEBUG' 50 # Enable or disable spider middlewares 51 # See http://scrapy.readthedocs.org/en/latest/topics/spider-middleware.html 52 #SPIDER_MIDDLEWARES = { 53 # 'TencentCrawlSpider.middlewares.TencentcrawlspiderSpiderMiddleware': 543, 54 #} 55 56 # Enable or disable downloader middlewares 57 # See http://scrapy.readthedocs.org/en/latest/topics/downloader-middleware.html 58 #DOWNLOADER_MIDDLEWARES = { 59 # 'TencentCrawlSpider.middlewares.MyCustomDownloaderMiddleware': 543, 60 #} 61 62 # Enable or disable extensions 63 # See http://scrapy.readthedocs.org/en/latest/topics/extensions.html 64 #EXTENSIONS = { 65 # 'scrapy.extensions.telnet.TelnetConsole': None, 66 #} 67 68 # Configure item pipelines 69 # See http://scrapy.readthedocs.org/en/latest/topics/item-pipeline.html 70 ITEM_PIPELINES = { 71 'TencentCrawlSpider.pipelines.TencentcrawlspiderPipeline': 300, 72 } 73 74 # Enable and configure the AutoThrottle extension (disabled by default) 75 # See http://doc.scrapy.org/en/latest/topics/autothrottle.html 76 #AUTOTHROTTLE_ENABLED = True 77 # The initial download delay 78 #AUTOTHROTTLE_START_DELAY = 5 79 # The maximum download delay to be set in case of high latencies 80 #AUTOTHROTTLE_MAX_DELAY = 60 81 # The average number of requests Scrapy should be sending in parallel to 82 # each remote server 83 #AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0 84 # Enable showing throttling stats for every response received: 85 #AUTOTHROTTLE_DEBUG = False 86 87 # Enable and configure HTTP caching (disabled by default) 88 # See http://scrapy.readthedocs.org/en/latest/topics/downloader-middleware.html#httpcache-middleware-settings 89 #HTTPCACHE_ENABLED = True 90 #HTTPCACHE_EXPIRATION_SECS = 0 91 #HTTPCACHE_DIR = 'httpcache' 92 #HTTPCACHE_IGNORE_HTTP_CODES = [] 93 #HTTPCACHE_STORAGE = 'scrapy.extensions.httpcache.FilesystemCacheStorage'
小Tips
1、python 爬蟲爬取內容時, \xa0 、 \u3000 的含義
\xa0 是不間斷空白符
我們通常所用的空格是 \x20 ,是在標准ASCII可見字符 0x20~0x7e 范圍內。
而 \xa0 屬於 latin1 (ISO/IEC_8859-1)中的擴展字符集字符,代表空白符nbsp(non-breaking space)。
latin1 字符集向下兼容 ASCII ( 0x20~0x7e )。通常我們見到的字符多數是 latin1 的,比如在 MySQL 數據庫中。
\u3000 是全角的空白符
根據Unicode編碼標准及其基本多語言面的定義, \u3000 屬於CJK字符的CJK標點符號區塊內,是空白字符之一。它的名字是 Ideographic Space ,有人譯作表意字空格、象形字空格等。顧名思義,就是全角的 CJK 空格。它跟 nbsp 不一樣,是可以被換行間斷的。常用於制造縮進, wiki 還說用於抬頭,但沒見過。
2、response.url # 獲取當前頁面url
3、在allow里面的正則匹配,有特殊字符('.','?')要加轉義字符'\'
page_lx = LinkExtractor(allow=('position\.php\?&start=\d+'))
4、字符串去空格 str.strip()
示例二:爬取網頁里面的信息(東莞)
爬蟲模塊
# -*- coding: utf-8 -*-
import scrapy
from scrapy.linkextractors import LinkExtractor
from scrapy.spiders import CrawlSpider, Rule
from newdongguan.items import NewdongguanItem
class DongdongSpider(CrawlSpider):
name = 'dongdong'
allowed_domains = ['wz.sun0769.com']
start_urls = ['http://wz.sun0769.com/index.php/question/questionType?type=4&page=']
# 每一頁的匹配規則
pagelink = LinkExtractor(allow=("type=4"))
# 每一頁里的每個帖子的匹配規則
contentlink = LinkExtractor(allow=(r"/html/question/\d+/\d+.shtml"))
rules = (
Rule(pagelink),
Rule(contentlink, callback = "parse_item",follow=False)
)
def parse_item(self, response):
item = NewdongguanItem()
# 標題
item['title'] = response.xpath('//div[contains(@class, "pagecenter p3")]//strong/text()').extract()[0]
# 編號
item['number'] = item['title'].split(' ')[-1].split(":")[-1]
# 內容,先使用有圖片情況下的匹配規則,如果有內容,返回所有內容的列表集合
content = response.xpath('//div[@class="contentext"]/text()').extract()
# 如果沒有內容,則返回空列表,則使用無圖片情況下的匹配規則
if len(content) == 0:
content = response.xpath('//div[@class="c1 text14_2"]/text()').extract()
item['content'] = "".join(content).strip()
else:
item['content'] = "".join(content).strip()
# 鏈接
item['url'] = response.url
yield item
管道模塊
# -*- coding: utf-8 -*-
# Define your item pipelines here
#
# Don't forget to add your pipeline to the ITEM_PIPELINES setting
# See: http://doc.scrapy.org/en/latest/topics/item-pipeline.html
import json
class DongguancrawlspiderPipeline(object):
def __init__(self):
self.file = open('dongguan.json','wb')
def process_item(self, item, spider):
text = json.dumps(dict(item),ensure_ascii=False)+'\n'
self.file.write(text.encode('utf-8'))
return item
def close_spider(self,spider):
self.file.close()
1 import scrapy 2 3 class DongguancrawlspiderItem(scrapy.Item): 4 # define the fields for your item here like: 5 title = scrapy.Field() 6 content = scrapy.Field() 7 url = scrapy.Field() 8 number = scrapy.Field()
1 # -*- coding: utf-8 -*- 2 3 # Scrapy settings for DongguanCrawlSpider project 4 # 5 # For simplicity, this file contains only settings considered important or 6 # commonly used. You can find more settings consulting the documentation: 7 # 8 # http://doc.scrapy.org/en/latest/topics/settings.html 9 # http://scrapy.readthedocs.org/en/latest/topics/downloader-middleware.html 10 # http://scrapy.readthedocs.org/en/latest/topics/spider-middleware.html 11 12 BOT_NAME = 'DongguanCrawlSpider' 13 14 SPIDER_MODULES = ['DongguanCrawlSpider.spiders'] 15 NEWSPIDER_MODULE = 'DongguanCrawlSpider.spiders' 16 17 18 # Crawl responsibly by identifying yourself (and your website) on the user-agent 19 #USER_AGENT = 'DongguanCrawlSpider (+http://www.yourdomain.com)' 20 21 # Obey robots.txt rules 22 # ROBOTSTXT_OBEY = True 23 24 # Configure maximum concurrent requests performed by Scrapy (default: 16) 25 #CONCURRENT_REQUESTS = 32 26 27 # Configure a delay for requests for the same website (default: 0) 28 # See http://scrapy.readthedocs.org/en/latest/topics/settings.html#download-delay 29 # See also autothrottle settings and docs 30 #DOWNLOAD_DELAY = 3 31 # The download delay setting will honor only one of: 32 #CONCURRENT_REQUESTS_PER_DOMAIN = 16 33 #CONCURRENT_REQUESTS_PER_IP = 16 34 35 # Disable cookies (enabled by default) 36 #COOKIES_ENABLED = False 37 38 # Disable Telnet Console (enabled by default) 39 #TELNETCONSOLE_ENABLED = False 40 41 # Override the default request headers: 42 DEFAULT_REQUEST_HEADERS = { 43 'User-Agent':'Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0;', 44 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', 45 # 'Accept-Language': 'en', 46 } 47 48 LOG_FILE = 'dongguan.log' 49 LOG_LEVER = 'DEBUG' 50 51 # Enable or disable spider middlewares 52 # See http://scrapy.readthedocs.org/en/latest/topics/spider-middleware.html 53 #SPIDER_MIDDLEWARES = { 54 # 'DongguanCrawlSpider.middlewares.DongguancrawlspiderSpiderMiddleware': 543, 55 #} 56 57 # Enable or disable downloader middlewares 58 # See http://scrapy.readthedocs.org/en/latest/topics/downloader-middleware.html 59 #DOWNLOADER_MIDDLEWARES = { 60 # 'DongguanCrawlSpider.middlewares.MyCustomDownloaderMiddleware': 543, 61 #} 62 63 # Enable or disable extensions 64 # See http://scrapy.readthedocs.org/en/latest/topics/extensions.html 65 #EXTENSIONS = { 66 # 'scrapy.extensions.telnet.TelnetConsole': None, 67 #} 68 69 # Configure item pipelines 70 # See http://scrapy.readthedocs.org/en/latest/topics/item-pipeline.html 71 ITEM_PIPELINES = { 72 'DongguanCrawlSpider.pipelines.DongguancrawlspiderPipeline': 300, 73 } 74 75 # Enable and configure the AutoThrottle extension (disabled by default) 76 # See http://doc.scrapy.org/en/latest/topics/autothrottle.html 77 #AUTOTHROTTLE_ENABLED = True 78 # The initial download delay 79 #AUTOTHROTTLE_START_DELAY = 5 80 # The maximum download delay to be set in case of high latencies 81 #AUTOTHROTTLE_MAX_DELAY = 60 82 # The average number of requests Scrapy should be sending in parallel to 83 # each remote server 84 #AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0 85 # Enable showing throttling stats for every response received: 86 #AUTOTHROTTLE_DEBUG = False 87 88 # Enable and configure HTTP caching (disabled by default) 89 # See http://scrapy.readthedocs.org/en/latest/topics/downloader-middleware.html#httpcache-middleware-settings 90 #HTTPCACHE_ENABLED = True 91 #HTTPCACHE_EXPIRATION_SECS = 0 92 #HTTPCACHE_DIR = 'httpcache' 93 #HTTPCACHE_IGNORE_HTTP_CODES = [] 94 #HTTPCACHE_STORAGE = 'scrapy.extensions.httpcache.FilesystemCacheStorage'
1、提取出來的鏈接可能被篡改,所以我們可以通過process_link來修改url(一般不會遇到)
import scrapy
from scrapy.linkextractors import LinkExtractor
from scrapy.spiders import CrawlSpider, Rule
from newdongguan.items import NewdongguanItem
class DongdongSpider(CrawlSpider):
name = 'dongdong'
allowed_domains = ['wz.sun0769.com']
start_urls = ['http://wz.sun0769.com/index.php/question/questionType?type=4&page=']
# 每一頁的匹配規則
pagelink = LinkExtractor(allow=("type=4"))
# 每一頁里的每個帖子的匹配規則
contentlink = LinkExtractor(allow=(r"/html/question/\d+/\d+.shtml"))
rules = (
# 本案例的url被web服務器篡改,需要調用process_links來處理提取出來的url
Rule(pagelink, process_links = "deal_links"),
Rule(contentlink, callback = "parse_item")
)
# links 是當前response里提取出來的鏈接列表
def deal_links(self, links):
for each in links:
each.url = each.url.replace("?","&").replace("Type&","Type?")
return links
def parse_item(self, response):
...
2、修改成spider類
# -*- coding: utf-8 -*-
import scrapy
from newdongguan.items import NewdongguanItem
class DongdongSpider(scrapy.Spider):
name = 'xixi'
allowed_domains = ['wz.sun0769.com']
url = 'http://wz.sun0769.com/index.php/question/questionType?type=4&page='
offset = 0
start_urls = [url + str(offset)]
def parse(self, response):
# 每一頁里的所有帖子的鏈接集合
links = response.xpath('//div[@class="greyframe"]/table//td/a[@class="news14"]/@href').extract()
# 迭代取出集合里的鏈接
for link in links:
# 提取列表里每個帖子的鏈接,發送請求放到請求隊列里,並調用self.parse_item來處理
yield scrapy.Request(link, callback = self.parse_item)
# 頁面終止條件成立前,會一直自增offset的值,並發送新的頁面請求,調用parse方法處理
if self.offset <= 71160:
self.offset += 30
# 發送請求放到請求隊列里,調用self.parse處理response
yield scrapy.Request(self.url + str(self.offset), callback = self.parse)
# 處理每個帖子的response內容
def parse_item(self, response):
item = NewdongguanItem()
# 標題
item['title'] = response.xpath('//div[contains(@class, "pagecenter p3")]//strong/text()').extract()[0]
# 編號
item['number'] = item['title'].split(' ')[-1].split(":")[-1]
# 內容,先使用有圖片情況下的匹配規則,如果有內容,返回所有內容的列表集合
content = response.xpath('//div[@class="contentext"]/text()').extract()
# 如果沒有內容,則返回空列表,則使用無圖片情況下的匹配規則
if len(content) == 0:
content = response.xpath('//div[@class="c1 text14_2"]/text()').extract()
item['content'] = "".join(content).strip()
else:
item['content'] = "".join(content).strip()
# 鏈接
item['url'] = response.url
# 交給管道
yield item
小Tips:
list = [a,b,c]
string = "123".join(list)
print(string)
>> a 123b 123c
string.replace("\xa0","") # 將空格換成空
string.strip() # 去首尾的空格
string.lstrip() # 去左邊(前面)的空格
string.rstrip() # 去右邊(后面)的空格
