Python3之urllib模塊

簡介

　　urllib是python的一個獲取url(Uniform Resource Locators，統一資源定位符)，可以用來抓取遠程的數據。

常用方法

（1）urlopen

　　urllib.request.urlopen(url, data=None,[timeout,]*,cafile=None,capath=None,cadefault=False,context=None)

urllib.request.urlopen() 可以獲取頁面，獲取頁面內容的數據格式為bytes類型，需要進行decode()解碼，轉換成str類型。

參數說明：

• url : 需要打開的網址
• data : 字典形式，默認為None時是GET方法，data不為空時, urlopen()的提交方式為POST，注意POST提交時，data需要轉換為字節;
• timeout : 設置網站訪問的超時時間

from urllib import request
response =  request.urlopen("http://members.3322.org/dyndns/getip")
# <http.client.HTTPResponse object at 0x031F63B0>
page = response.read()
# b'106.37.169.186\n'
page = page.decode("utf-8")
# '106.37.169.186\n'

# 使用with語句
with request.urlopen("http://members.3322.org/dyndns/getip") as response:
    page = response.read()
    print(page.decode("utf-8"))

　　注意：urllib.request 使用相同的接口來處理所有類型的url，比如：

req = urllib.request.urlopen('ftp://example.com/')

　　

　　urlopen返回對象提供的方法：

• read(),readline(),readlines(),fileno(),close() : 對HTTPResponse類型數據進行操作
• info() : 返回HTTPMessage對象，表示遠程服務器返回的頭信息
• getcode() : 返回HTTP狀態碼，如果是http請求，200請求成功完成，404網頁未找到
• geturl(): 返回請求的url

 

（2）Request

　　urllib.request.Request(url,data=None,headers={},method=None)

from urllib import request

url = r'http://www.lagou.com/zhaopin/Python/?labelWords=label'
headers = {
    'User-Agent': r'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) '
                  r'Chrome/45.0.2454.85 Safari/537.36 115Browser/6.0.3',
    'Referer': r'http://www.lagou.com/zhaopin/Python/?labelWords=label',
    'Connection': 'keep-alive'
}
req = request.Request(url, headers=headers)
page = request.urlopen(req).read()
page = page.decode('utf-8')

　　

（3）parse.urlencode

　　urllib.parse.urlencode(query, doseq=False,safe='',encoding=None,errors=None)

urlencode()的主要作用就是將url附上要提交的數據. 對data數據進行編碼。

from urllib import request, parse
url = r'http://www.lagou.com/jobs/positionAjax.json?'
headers = {
    'User-Agent': r'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) '
                  r'Chrome/45.0.2454.85 Safari/537.36 115Browser/6.0.3',
    'Referer': r'http://www.lagou.com/zhaopin/Python/?labelWords=label',
    'Connection': 'keep-alive'
}
data = {
    'first': 'true',
    'pn': 1,
    'kd': 'Python'
}
data = parse.urlencode(data).encode('utf-8')
# 此時data是字節 b'first=true&pn=1&kd=Python' ，POST的數據必須是bytes或者iterable of bytes，不能是str，因此需要encode編碼
# 經過urlencode轉換后的data數據為'first=true&pn=1&kd=Python'
# 最后提交的url為：http://www.lagou.com/jobs/positionAjax.json?first=true?pn=1?kd=Python
req = request.Request(url, headers=headers, data=data)
# 此時req : <urllib.request.Request object at 0x02F52A30>
page = request.urlopen(req).read()
# 此時page是字節： b'{"success":false,"msg":"\xe6\x82\xa8\xe6\x93\x8d\xe4\xbd\x9c\xe5\xa4\xaa\xe9\xa2\x91\xe7\xb9\x81,\xe8\xaf\xb7\xe7\xa8\x8d\xe5\x90\x8e\xe5\x86\x8d\xe8\xae\xbf\xe9\x97\xae","clientIp":"106.37.169.186"}\n
page = page.decode('utf-8')
# 此時page是字符串："success":false,"msg":"您操作太頻繁,請稍后再訪問","clientIp":"106.37.169.186"}

　　

（4）代理 request.ProxyHandler(proxies=None)

當需要抓取的網站設置了訪問限制，這時就需要用到代理來抓取數據。

from urllib import request, parse
data = {
        'first': 'true',
        'pn': 1,
        'kd': 'Python'
    }
proxy = request.ProxyHandler({'http': '5.22.195.215:80'})  # 設置proxy
opener = request.build_opener(proxy)  # 掛載opener
request.install_opener(opener)  # 安裝opener
data = parse.urlencode(data).encode('utf-8')
page = opener.open(url, data).read()
page = page.decode('utf-8')
return page

　　

（5）異常處理

　　urlopen在不能處理某個響應的時候會拋出URLError， HTTPError是URLError的子類，在遇到HTTP URL的特殊情況時被拋出。異常類來自於 urllib.error 模塊。

URLError : 

　　一般來說，URLError被拋出是因為沒有網絡連接(沒有到指定服務器的路徑)，或者是指定服務器不存在。在這種情況下，拋出的異常將會包含一個‘reason’屬性，
這是包含一個錯誤碼和一段錯誤信息的元組.

req = urllib.request.Request('http://www.pretend_server.org')
try: 
	urllib.request.urlopen(req)
except urllib.error.URLError as e:
    print(e.reason)      
# 輸出
(4, 'getaddrinfo failed')

　　

HTTPError :
　　每一個來自服務器的HTTP響應都包含一個數字的“狀態碼”。有時狀態碼表明服務器不能執行請求。默認的處理程序會為你處理其中的部分響應(比如，如果響應是“重定向”，
要求客戶端從一個不同的URL中獲取資料，那么urllib將會為你處理這個)。對於那些不能處理的響應，urlopen將會拋出一個HTTPError。
典型的錯誤包括‘404’(頁面未找到)，‘403’(請求禁止)，和‘401’(請求認證)。

# Table mapping response codes to messages; entries have the
# form {code: (shortmessage, longmessage)}.
responses = {
    100: ('Continue', 'Request received, please continue'),
    101: ('Switching Protocols',
          'Switching to new protocol; obey Upgrade header'),

    200: ('OK', 'Request fulfilled, document follows'),
    201: ('Created', 'Document created, URL follows'),
    202: ('Accepted',
          'Request accepted, processing continues off-line'),
    203: ('Non-Authoritative Information', 'Request fulfilled from cache'),
    204: ('No Content', 'Request fulfilled, nothing follows'),
    205: ('Reset Content', 'Clear input form for further input.'),
    206: ('Partial Content', 'Partial content follows.'),

    300: ('Multiple Choices',
          'Object has several resources -- see URI list'),
    301: ('Moved Permanently', 'Object moved permanently -- see URI list'),
    302: ('Found', 'Object moved temporarily -- see URI list'),
    303: ('See Other', 'Object moved -- see Method and URL list'),
    304: ('Not Modified',
          'Document has not changed since given time'),
    305: ('Use Proxy',
          'You must use proxy specified in Location to access this '
          'resource.'),
    307: ('Temporary Redirect',
          'Object moved temporarily -- see URI list'),

    400: ('Bad Request',
          'Bad request syntax or unsupported method'),
    401: ('Unauthorized',
          'No permission -- see authorization schemes'),
    402: ('Payment Required',
          'No payment -- see charging schemes'),
    403: ('Forbidden',
          'Request forbidden -- authorization will not help'),
    404: ('Not Found', 'Nothing matches the given URI'),
    405: ('Method Not Allowed',
          'Specified method is invalid for this server.'),
    406: ('Not Acceptable', 'URI not available in preferred format.'),
    407: ('Proxy Authentication Required', 'You must authenticate with '
          'this proxy before proceeding.'),
    408: ('Request Timeout', 'Request timed out; try again later.'),
    409: ('Conflict', 'Request conflict.'),
    410: ('Gone',
          'URI no longer exists and has been permanently removed.'),
    411: ('Length Required', 'Client must specify Content-Length.'),
    412: ('Precondition Failed', 'Precondition in headers is false.'),
    413: ('Request Entity Too Large', 'Entity is too large.'),
    414: ('Request-URI Too Long', 'URI is too long.'),
    415: ('Unsupported Media Type', 'Entity body in unsupported format.'),
    416: ('Requested Range Not Satisfiable',
          'Cannot satisfy request range.'),
    417: ('Expectation Failed',
          'Expect condition could not be satisfied.'),

    500: ('Internal Server Error', 'Server got itself in trouble'),
    501: ('Not Implemented',
          'Server does not support this operation'),
    502: ('Bad Gateway', 'Invalid responses from another server/proxy.'),
    503: ('Service Unavailable',
          'The server cannot process the request due to a high load'),
    504: ('Gateway Timeout',
          'The gateway server did not receive a timely response'),
    505: ('HTTP Version Not Supported', 'Cannot fulfill request.'),
    }

　　

　　異常處理方式：

req = urllib.request.Request('http://www.python.org/fish.html')
try:
    urllib.request.urlopen(req)
except urllib.error.HTTPError as e:
    print (e.code)
    print (e.info())
    print (e.geturl())
    print (e.read())

　　或者：

from urllib.request import Request, urlopen
from urllib.error import URLError
req = Request(someurl)
try:
    response = urlopen(req)
except URLError as e:
    if hasattr(e, 'reason'):
        print('We failed to reach a server.')
        print('Reason: ', e.reason)
    elif hasattr(e, 'code'):
        print('The server couldn\'t fulfill the request.')
        print('Error code: ', e.code)