【Python】 requests 爬取博客園內容AttributeError: 'NoneType' object has no attribute 'xpath'

本篇文章主要介紹requests獲取網頁內容出現 'NoneType' object has no attribute 'xpath' 異常的解決思路

下面是出錯的代碼：

import requests
from lxml import etree
response = requests.get('https://blog.csdn.net/it_xf?viewmode=contents')
etree_html = etree.HTML(response.text)
content = etree_html.xpath('//*[@id="mainBox"]/main/div[2]/div[1]/h4/a/text()')

for each in content:
    replace = each.replace('\n', '').replace(' ', '')
    if replace == '\n' or replace == '':
        continue
    else:
        print(replace)　　

1、錯誤分析

獲取到的html.text 為 空字符串；所以下面拋出異常NoneType 

原因是請求Get 需要增加 headers來解決反扒；模擬瀏覽器請求來獲取數據；

2、解決辦法

首先找到需要的headers，headers 如何尋找？看下圖的標記：

 

然后直接把上面的headers復制出來放到代碼中進行改造；

改造后的代碼如下：

 

import requests
from lxml import etree
headers = {
    'user-agent':'Mozilla/5.0 (Windows NT 6.1; WOW64) '
                 'AppleWebKit/537.36 (KHTML, like Gecko) '
                 'Chrome/58.0.3029.110 Safari/537.36 SE 2.X MetaSr 1.0'
}
response = requests.get('https://blog.csdn.net/it_xf?viewmode=contents', headers = headers)
etree_html = etree.HTML(response.text)
content = etree_html.xpath('//*[@id="mainBox"]/main/div[2]/div[1]/h4/a/text()')

for each in content:
    replace = each.replace('\n', '').replace(' ', '')
    if replace == '\n' or replace == '':
        continue
    else:
        print(replace)