老猿前期執行如下代碼時報“‘utf-8’ codec can’t decode byte”錯，代碼及錯誤信息如下：
>>> import urllib.request
>>> def mkhead():
header = {'Accept':'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3',
'Accept-Encoding':'gzip',
'Accept-Language':'zh-CN,zh;q=0.9',
'Connection':'keep-alive',
'User-Agent':'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36'}
return header
>>> def readweb(site):
header = mkhead()
try:
req = urllib.request.Request(url=site,headers=header)
text = urllib.request.urlopen(req).read().decode()
except Exception as e:
print(e)
return None
else:return text
>>> readweb(r'https://blog.csdn.net/LaoYuanPython')
'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte
>>>
才開始以為是decode編碼的問題，試了gbk等方式還是不行，最后發現是因為http請求報文頭“‘Accept-Encoding’:‘gzip’”導致服務器返回的報文壓縮了，把這個報文頭信息去掉再執行就ok了，如下：
>>> import urllib.request
>>> def mkhead():
header = {'Accept':'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3',
'Accept-Language':'zh-CN,zh;q=0.9',
'Connection':'keep-alive',
'User-Agent':'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36'}
return header
>>> def readweb(site):
header = mkhead()
try:
req = urllib.request.Request(url=site,headers=header)
text = urllib.request.urlopen(req).read().decode()
except Exception as e:
print(e)
return None
else:return text
>>> readweb(r'https://blog.csdn.net/LaoYuanPython')
Squeezed text(273 lines)
>>> readweb(r'https://blog.csdn.net/LaoYuanPython')[0:100]
'<!DOCTYPE html>\n<html lang="zh-CN">\n<head>\n <meta charset="UTF-8">\n <link rel="canonical" href'
>>>
如果希望處理壓縮報文，請參考《第14.7節 Python模擬瀏覽器訪問實現http報文體壓縮傳輸》。
老猿Python，跟老猿學Python!
博客地址：https://blog.csdn.net/LaoYuanPython
老猿Python博客文章目錄：https://blog.csdn.net/LaoYuanPython/article/details/98245036
請大家多多支持，點贊、評論和加關注！謝謝！