一、話說爬蟲
先說說爬蟲，爬蟲常被用來抓取特定網站網頁的HTML數據，定位在后端數據的獲取,而對於網站而言，爬蟲給網站帶來流量的同時，一些設計不好的爬蟲由於爬得太猛，導致給網站來帶很大的負擔，當然再加上一些網站並不希望被爬取，所以就出現了許許多多的反爬技術。
二、安裝模塊
1. requests
模塊安裝方法：
pip3 install requests
2、beautisoup模塊
軟件安裝方法：
pip3 install beautifulsoup4 或 pip3 install bs4
3、lxml模塊
#必須先安裝whell依賴 （請換成國內pip源進行安裝,否則容易報錯）
pip install wheel
#在cmd中，輸入python進入python。
然后輸入import pip;print(pip.pep425tags.get_supported())，界面上輸出當前python的版本信息，如圖。
再跟據上面查到的版本信息，找到下面對應的版本進行安裝。
#下載地址：https://pypi.python.org/pypi/lxml/3.7.3 (網站打不開，請翻牆，就可以打開)
#python3.5就選擇cp3m版本 lxml-3.7.3-cp35-cp35m-win32.whl
#安裝方法
pip3 install lxml-3.6.4-cp35-cp35m-win_amd64.whl
進入python3，輸入import lxml，未報錯，即表示安裝成功。
三、requests模塊用法
Python標准庫中提供了：urllib、urllib2、httplib等模塊以供Http請求，但是，它的 API 太渣了。它是為另一個時代、另一個互聯網所創建的。它需要巨量的工作，甚至包括各種方法覆蓋，來完成最簡單的任務。
Requests 是使用 Apache2 Licensed 許可證的 基於Python開發的HTTP 庫，其在Python內置模塊的基礎上進行了高度的封裝，從而使得Pythoner進行網絡請求時，變得美好了許多，使用Requests可以輕而易舉的完成瀏覽器可有的任何操作。
1、GET請求
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
|
# 1、無參數實例
import requests
ret = requests.get('https://github.com/timeline.json')
print ret.url
print ret.text
# 2、有參數實例
import requests
payload = {'key1': 'value1', 'key2': 'value2'}
ret = requests.get("http://httpbin.org/get", params=payload)
print ret.url
print ret.text
|
2、POST請求
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
|
# 1、基本POST實例
import requests
payload = {'key1': 'value1', 'key2': 'value2'}
ret = requests.post("http://httpbin.org/post", data=payload)
print ret.text
# 2、發送請求頭和數據實例
import requests
import json
url = 'https://api.github.com/some/endpoint'
payload = {'some': 'data'}
headers = {'content-type': 'application/json'}
ret = requests.post(url, data=json.dumps(payload), headers=headers)
print ret.text
print ret.cookies
|
3、requests屬性
response = requests.get('URL') response.text # 獲取文本內容 response.content # 獲取文本內容，字節 response.encoding # 設置返回結果的編碼 response.aparent_encoding # 獲取網站原始的編碼 response.status_code # 狀態碼 response.cookies.get_dict() # cookies
4、關系和方法
- 方法關系
requests.get(url, params=None, **kwargs)
requests.post(url, data=None, json=None, **kwargs)
requests.put(url, data=None, **kwargs)
requests.head(url, **kwargs)
requests.delete(url, **kwargs)
requests.patch(url, data=None, **kwargs)
requests.options(url, **kwargs)
- 在此方法的基礎上構建
requests.request(method, url, **kwargs)
- method: 提交方式 - url: 提交地址 - params: 在URL中傳遞的參數,GET requests.request( method='GET', url= 'http://www.nulige.com', params = {'k1':'v1','k2':'v2'} ) # http://www.nulige.com?k1=v1&k2=v2
- data: 在請求體里傳遞的數據 requests.request( method='POST', url= 'http://www.nulige.com', params = {'k1':'v1','k2':'v2'}, data = {'use':'alex','pwd': '123','x':[11,2,3} ) 請求頭: content-type: application/url-form-encod..... 請求體: use=alex&pwd=123
- json 在請求體里傳遞的數據 requests.request( method='POST', url= 'http://www.oldboyedu.com', params = {'k1':'v1','k2':'v2'}, json = {'use':'alex','pwd': '123'} ) 請求頭: content-type: application/json 請求體: "{'use':'alex','pwd': '123'}" PS: 字典中嵌套字典時 - headers 請求頭 requests.request( method='POST', url= 'http://www.oldboyedu.com', params = {'k1':'v1','k2':'v2'}, json = {'use':'alex','pwd': '123'}, headers={ 'Referer': 'http://dig.chouti.com/', 'User-Agent': "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36" } )
- cookies Cookies
- files 上傳文件
- auth 基本認證(headers中加入加密的用戶名和密碼)
- timeout 請求和響應的超時時間
- allow_redirects 是否允許重定向
- proxies 代理 （nginx反向代理模塊）
- verify 是否忽略證書
- cert 證書文件
- stream 流的方式迭代下載
- session: 用於保存客戶端歷史訪問信息
參數用法示例：
def param_method_url(): # requests.request(method='get', url='http://127.0.0.1:8000/test/') # requests.request(method='post', url='http://127.0.0.1:8000/test/') pass def param_param(): # - 可以是字典 # - 可以是字符串 # - 可以是字節（ascii編碼以內） # requests.request(method='get', # url='http://127.0.0.1:8000/test/', # params={'k1': 'v1', 'k2': '水電費'}) # requests.request(method='get', # url='http://127.0.0.1:8000/test/', # params="k1=v1&k2=水電費&k3=v3&k3=vv3") # requests.request(method='get', # url='http://127.0.0.1:8000/test/', # params=bytes("k1=v1&k2=k2&k3=v3&k3=vv3", encoding='utf8')) # 錯誤 # requests.request(method='get', # url='http://127.0.0.1:8000/test/', # params=bytes("k1=v1&k2=水電費&k3=v3&k3=vv3", encoding='utf8')) pass def param_data(): # 可以是字典 # 可以是字符串 # 可以是字節 # 可以是文件對象 # requests.request(method='POST', # url='http://127.0.0.1:8000/test/', # data={'k1': 'v1', 'k2': '水電費'}) # requests.request(method='POST', # url='http://127.0.0.1:8000/test/', # data="k1=v1; k2=v2; k3=v3; k3=v4" # ) # requests.request(method='POST', # url='http://127.0.0.1:8000/test/', # data="k1=v1;k2=v2;k3=v3;k3=v4", # headers={'Content-Type': 'application/x-www-form-urlencoded'} # ) # requests.request(method='POST', # url='http://127.0.0.1:8000/test/', # data=open('data_file.py', mode='r', encoding='utf-8'), # 文件內容是：k1=v1;k2=v2;k3=v3;k3=v4 # headers={'Content-Type': 'application/x-www-form-urlencoded'} # ) pass def param_json(): # 將json中對應的數據進行序列化成一個字符串，json.dumps(...) # 然后發送到服務器端的body中，並且Content-Type是 {'Content-Type': 'application/json'} requests.request(method='POST', url='http://127.0.0.1:8000/test/', json={'k1': 'v1', 'k2': '水電費'}) def param_headers(): # 發送請求頭到服務器端 requests.request(method='POST', url='http://127.0.0.1:8000/test/', json={'k1': 'v1', 'k2': '水電費'}, headers={'Content-Type': 'application/x-www-form-urlencoded'} ) def param_cookies(): # 發送Cookie到服務器端 requests.request(method='POST', url='http://127.0.0.1:8000/test/', data={'k1': 'v1', 'k2': 'v2'}, cookies={'cook1': 'value1'}, ) # 也可以使用CookieJar（字典形式就是在此基礎上封裝） from http.cookiejar import CookieJar from http.cookiejar import Cookie obj = CookieJar() obj.set_cookie(Cookie(version=0, name='c1', value='v1', port=None, domain='', path='/', secure=False, expires=None, discard=True, comment=None, comment_url=None, rest={'HttpOnly': None}, rfc2109=False, port_specified=False, domain_specified=False, domain_initial_dot=False, path_specified=False) ) requests.request(method='POST', url='http://127.0.0.1:8000/test/', data={'k1': 'v1', 'k2': 'v2'}, cookies=obj) def param_files(): # 發送文件 # file_dict = { # 'f1': open('readme', 'rb') # } # requests.request(method='POST', # url='http://127.0.0.1:8000/test/', # files=file_dict) # 發送文件，定制文件名 # file_dict = { # 'f1': ('test.txt', open('readme', 'rb')) # } # requests.request(method='POST', # url='http://127.0.0.1:8000/test/', # files=file_dict) # 發送文件，定制文件名 # file_dict = { # 'f1': ('test.txt', "hahsfaksfa9kasdjflaksdjf") # } # requests.request(method='POST', # url='http://127.0.0.1:8000/test/', # files=file_dict) # 發送文件，定制文件名 # file_dict = { # 'f1': ('test.txt', "hahsfaksfa9kasdjflaksdjf", 'application/text', {'k1': '0'}) # } # requests.request(method='POST', # url='http://127.0.0.1:8000/test/', # files=file_dict) pass def param_auth(): from requests.auth import HTTPBasicAuth, HTTPDigestAuth ret = requests.get('https://api.github.com/user', auth=HTTPBasicAuth('wupeiqi', 'sdfasdfasdf')) print(ret.text) # ret = requests.get('http://192.168.1.1', # auth=HTTPBasicAuth('admin', 'admin')) # ret.encoding = 'gbk' # print(ret.text) # ret = requests.get('http://httpbin.org/digest-auth/auth/user/pass', auth=HTTPDigestAuth('user', 'pass')) # print(ret) # def param_timeout(): # ret = requests.get('http://google.com/', timeout=1) # print(ret) # ret = requests.get('http://google.com/', timeout=(5, 1)) # print(ret) pass def param_allow_redirects(): ret = requests.get('http://127.0.0.1:8000/test/', allow_redirects=False) print(ret.text) def param_proxies(): # proxies = { # "http": "61.172.249.96:80", # "https": "http://61.185.219.126:3128", # } # proxies = {'http://10.20.1.128': 'http://10.10.1.10:5323'} # ret = requests.get("http://www.proxy360.cn/Proxy", proxies=proxies) # print(ret.headers) # from requests.auth import HTTPProxyAuth # # proxyDict = { # 'http': '77.75.105.165', # 'https': '77.75.105.165' # } # auth = HTTPProxyAuth('username', 'mypassword') # # r = requests.get("http://www.google.com", proxies=proxyDict, auth=auth) # print(r.text) pass def param_stream(): ret = requests.get('http://127.0.0.1:8000/test/', stream=True) print(ret.content) ret.close() # from contextlib import closing # with closing(requests.get('http://httpbin.org/get', stream=True)) as r: # # 在此處理響應。 # for i in r.iter_content(): # print(i) def requests_session(): import requests session = requests.Session() ### 1、首先登陸任何頁面，獲取cookie i1 = session.get(url="http://dig.chouti.com/help/service") ### 2、用戶登陸，攜帶上一次的cookie，后台對cookie中的 gpsd 進行授權 i2 = session.post( url="http://dig.chouti.com/login", data={ 'phone': "8615131255089", 'password': "xxxxxx", 'oneMonth': "" } ) i3 = session.post( url="http://dig.chouti.com/link/vote?linksId=8589623", ) print(i3.text)
參考：http://cn.python-requests.org/zh_CN/latest/user/quickstart.html#id4
四、BeautifulSoup
該模塊用於接收一個HTML或XML字符串，然后將其進行格式化，之后遍可以使用他提供的方法進行快速查找指定元素，從而使得在HTML或XML中查找指定元素變得簡單。
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
|
from
bs4
import
BeautifulSoup
html_doc
=
"""
<html><head><title>The Dormouse's story</title></head>
<body>
asdf
<div class="title">
<b>The Dormouse's story總共</b>
<h1>f</h1>
</div>
<div class="story">Once upon a time there were three little sisters; and their names were
<a class="sister0" id="link1">Els<span>f</span>ie</a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</div>
ad<br/>sf
<p class="story">...</p>
</body>
</html>
"""
soup
=
BeautifulSoup(html_doc, features
=
"lxml"
)
# 找到第一個a標簽
tag1
=
soup.find(name
=
'a'
)
# 找到所有的a標簽
tag2
=
soup.find_all(name
=
'a'
)
# 找到id＝link2的標簽
tag3
=
soup.select(
'#link2'
)
|
使用示例：
|
1
2
3
4
5
6
7
8
9
10
11
|
from
bs4
import
BeautifulSoup
html_doc
=
"""
<html><head><title>The Dormouse's story</title></head>
<body>
...
</body>
</html>
"""
soup
=
BeautifulSoup(html_doc, features
=
"lxml"
)
|
1. name，標簽名稱
|
1
2
3
4
5
|
# tag = soup.find('a')
# name = tag.name # 獲取
# print(name)
# tag.name = 'span' # 設置
# print(soup)
|
2. attr，標簽屬性
|
1
2
3
4
5
6
|
# tag = soup.find('a')
# attrs = tag.attrs # 獲取
# print(attrs)
# tag.attrs = {'ik':123} # 設置
# tag.attrs['id'] = 'iiiii' # 設置
# print(soup)
|
3. children,所有子標簽
|
1
2
|
# body = soup.find('body')
# v = body.children
|
4. children,所有子子孫孫標簽
|
1
2
|
# body = soup.find('body')
# v = body.descendants
|
5. clear,將標簽的所有子標簽全部清空（保留標簽名）
|
1
2
3
|
# tag = soup.find('body')
# tag.clear()
# print(soup)
|
6. decompose,遞歸的刪除所有的標簽
|
1
2
3
|
# body = soup.find('body')
# body.decompose()
# print(soup)
|
7. extract,遞歸的刪除所有的標簽，並獲取刪除的標簽
|
1
2
3
|
# body = soup.find('body')
# v = body.extract()
# print(soup)
|
8. decode,轉換為字符串（含當前標簽）；decode_contents（不含當前標簽）
|
1
2
3
4
|
# body = soup.find('body')
# v = body.decode()
# v = body.decode_contents()
# print(v)
|
9. encode,轉換為字節（含當前標簽）；encode_contents（不含當前標簽）
|
1
2
3
4
|
# body = soup.find('body')
# v = body.encode()
# v = body.encode_contents()
# print(v)
|
10. find,獲取匹配的第一個標簽
|
1
2
3
4
5
|
# tag = soup.find('a')
# print(tag)
# tag = soup.find(name='a', attrs={'class': 'sister'}, recursive=True, text='Lacie')
# tag = soup.find(name='a', class_='sister', recursive=True, text='Lacie')
# print(tag)
|
11. find_all,獲取匹配的所有標簽
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
|
# tags = soup.find_all('a')
# print(tags)
# tags = soup.find_all('a',limit=1)
# print(tags)
# tags = soup.find_all(name='a', attrs={'class': 'sister'}, recursive=True, text='Lacie')
# # tags = soup.find(name='a', class_='sister', recursive=True, text='Lacie')
# print(tags)
# ####### 列表 #######
# v = soup.find_all(name=['a','div'])
# print(v)
# v = soup.find_all(class_=['sister0', 'sister'])
# print(v)
# v = soup.find_all(text=['Tillie'])
# print(v, type(v[0]))
# v = soup.find_all(id=['link1','link2'])
# print(v)
# v = soup.find_all(href=['link1','link2'])
# print(v)
# ####### 正則 #######
import
re
# rep = re.compile('p')
# rep = re.compile('^p')
# v = soup.find_all(name=rep)
# print(v)
# rep = re.compile('sister.*')
# v = soup.find_all(class_=rep)
# print(v)
# rep = re.compile('http://www.oldboy.com/static/.*')
# v = soup.find_all(href=rep)
# print(v)
# ####### 方法篩選 #######
# def func(tag):
# return tag.has_attr('class') and tag.has_attr('id')
# v = soup.find_all(name=func)
# print(v)
# ## get,獲取標簽屬性
# tag = soup.find('a')
# v = tag.get('id')
# print(v)
|
12. has_attr,檢查標簽是否具有該屬性
|
1
2
3
|
# tag = soup.find('a')
# v = tag.has_attr('id')
# print(v)
|
13. get_text,獲取標簽內部文本內容
|
1
2
3
|
# tag = soup.find('a')
# v = tag.get_text
# print(v)
|
14. index,檢查標簽在某標簽中的索引位置
|
1
2
3
4
5
6
7
|
# tag = soup.find('body')
# v = tag.index(tag.find('div'))
# print(v)
# tag = soup.find('body')
# for i,v in enumerate(tag):
# print(i,v)
|
15. is_empty_element,是否是空標簽(是否可以是空)或者自閉合標簽，
判斷是否是如下標簽：'br' , 'hr', 'input', 'img', 'meta','spacer', 'link', 'frame', 'base'
|
1
2
3
|
# tag = soup.find('br')
# v = tag.is_empty_element
# print(v)
|
16. 當前的關聯標簽
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
|
# soup.next
# soup.next_element
# soup.next_elements
# soup.next_sibling
# soup.next_siblings
#
# tag.previous
# tag.previous_element
# tag.previous_elements
# tag.previous_sibling
# tag.previous_siblings
#
# tag.parent
# tag.parents
|
17. 查找某標簽的關聯標簽
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
|
# tag.find_next(...)
# tag.find_all_next(...)
# tag.find_next_sibling(...)
# tag.find_next_siblings(...)
# tag.find_previous(...)
# tag.find_all_previous(...)
# tag.find_previous_sibling(...)
# tag.find_previous_siblings(...)
# tag.find_parent(...)
# tag.find_parents(...)
# 參數同find_all
|
18. select,select_one, CSS選擇器
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
|
soup.select(
"title"
)
soup.select(
"p nth-of-type(3)"
)
soup.select(
"body a"
)
soup.select(
"html head title"
)
tag
=
soup.select(
"span,a"
)
soup.select(
"head > title"
)
soup.select(
"p > a"
)
soup.select(
"p > a:nth-of-type(2)"
)
soup.select(
"p > #link1"
)
soup.select(
"body > a"
)
soup.select(
"#link1 ~ .sister"
)
soup.select(
"#link1 + .sister"
)
soup.select(
".sister"
)
soup.select(
"[class~=sister]"
)
soup.select(
"#link1"
)
soup.select(
"a#link2"
)
soup.select(
'a[href]'
)
soup.select(
'a[href="http://example.com/elsie"]'
)
soup.select(
'a[href^="http://example.com/"]'
)
soup.select(
'a[href$="tillie"]'
)
soup.select(
'a[href*=".com/el"]'
)
from
bs4.element
import
Tag
def
default_candidate_generator(tag):
for
child
in
tag.descendants:
if
not
isinstance
(child, Tag):
continue
if
not
child.has_attr(
'href'
):
continue
yield
child
tags
=
soup.find(
'body'
).select(
"a"
, _candidate_generator
=
default_candidate_generator)
print
(
type
(tags), tags)
from
bs4.element
import
Tag
def
default_candidate_generator(tag):
for
child
in
tag.descendants:
if
not
isinstance
(child, Tag):
continue
if
not
child.has_attr(
'href'
):
continue
yield
child
tags
=
soup.find(
'body'
).select(
"a"
, _candidate_generator
=
default_candidate_generator, limit
=
1
)
print
(
type
(tags), tags)
|
19. 標簽的內容
|
1
2
3
4
5
6
7
8
9
10
11
12
13
|
# tag = soup.find('span')
# print(tag.string) # 獲取
# tag.string = 'new content' # 設置
# print(soup)
# tag = soup.find('body')
# print(tag.string)
# tag.string = 'xxx'
# print(soup)
# tag = soup.find('body')
# v = tag.stripped_strings # 遞歸內部獲取所有標簽的文本
# print(v)
|
20.append在當前標簽內部追加一個標簽
|
1
2
3
4
5
6
7
8
9
10
|
# tag = soup.find('body')
# tag.append(soup.find('a'))
# print(soup)
#
# from bs4.element import Tag
# obj = Tag(name='i',attrs={'id': 'it'})
# obj.string = '我是一個新來的'
# tag = soup.find('body')
# tag.append(obj)
# print(soup)
|
21.insert在當前標簽內部指定位置插入一個標簽
|
1
2
3
4
5
6
|
# from bs4.element import Tag
# obj = Tag(name='i', attrs={'id': 'it'})
# obj.string = '我是一個新來的'
# tag = soup.find('body')
# tag.insert(2, obj)
# print(soup)
|
22. insert_after,insert_before 在當前標簽后面或前面插入
|
1
2
3
4
5
6
7
|
# from bs4.element import Tag
# obj = Tag(name='i', attrs={'id': 'it'})
# obj.string = '我是一個新來的'
# tag = soup.find('body')
# # tag.insert_before(obj)
# tag.insert_after(obj)
# print(soup)
|
23. replace_with 在當前標簽替換為指定標簽
|
1
2
3
4
5
6
|
# from bs4.element import Tag
# obj = Tag(name='i', attrs={'id': 'it'})
# obj.string = '我是一個新來的'
# tag = soup.find('div')
# tag.replace_with(obj)
# print(soup)
|
24. 創建標簽之間的關系
|
1
2
3
4
|
# tag = soup.find('div')
# a = soup.find('a')
# tag.setup(previous_sibling=a)
# print(tag.previous_sibling)
|
25. wrap，將指定標簽把當前標簽包裹起來
|
1
2
3
4
5
6
7
8
9
10
11
|
# from bs4.element import Tag
# obj1 = Tag(name='div', attrs={'id': 'it'})
# obj1.string = '我是一個新來的'
#
# tag = soup.find('a')
# v = tag.wrap(obj1)
# print(soup)
# tag = soup.find('a')
# v = tag.wrap(soup.find('p'))
# print(soup)
|
26. unwrap，去掉當前標簽，將保留其包裹的標簽
|
1
2
3
|
# tag = soup.find('a')
# v = tag.unwrap()
# print(soup)
|
更多參數官方：http://beautifulsoup.readthedocs.io/zh_CN/v4.4.0/
五、示例
把下面代碼，加入到代碼中，可以下載網站源碼到本地分析
with open('weixin.html','wb') as f:
f.write(wx_login_page.content)
1、爬取汽車之家新聞頻道頁面里面的圖片
#!/usr/bin/env python
# -*- coding:utf-8 -*- # Author: nulige import requests from bs4 import BeautifulSoup response = requests.get( url='http://www.autohome.com.cn/news/' ) #解決爬蟲亂碼問題 response.encoding = response.apparent_encoding # 生成Soup對象， soup = BeautifulSoup(response.text, features='html.parser') # find查找第一個符合條件的對象 target = soup.find(id='auto-channel-lazyload-article') #find_all查找所有符合的對象，查找出來的值在列表中 li_list = target.find_all('li') #循環拿到具體每個對象 for i in li_list: a = i.find('a') if a: print(a.attrs.get('href')) # # .attrs查找到屬性 txt = a.find('h3').text # 是對象 img_url = a.find('img').attrs.get('src') print(img_url) # 再發一個請求 img_response = requests.get(url=img_url) import uuid file_name = str(uuid.uuid4()) + '.jpg' with open(file_name,'wb') as f: f.write(img_response.content)
備注：
# 找到第一個a標簽
tag1
=
soup.find(name
=
'a'
)
# 找到所有的a標簽
tag2
=
soup.find_all(name
=
'a'
)
# 找到id＝link2的標簽
tag3
=
soup.select(
'#link2'
)
2、自動登陸抽屜網
#!/usr/bin/env python # -*- coding: utf8 -*- # __Author: "Skiler Hao" # date: 2017/5/10 11:06 import requests from bs4 import BeautifulSoup # 第一次請求 first_request_response = requests.get( url = 'http://dig.chouti.com/', ) # 獲取第一次登錄獲取的cookie內容 firstget_cookie_dict = first_request_response.cookies.get_dict() # 登錄POST請求 post_dict = { 'phone': '8618811*****', #86+手機號碼 'password': '******', #密碼 'oneMonth': 1 } # 發送請求，攜帶cookie和數據 login_response = requests.post( url = 'http://dig.chouti.com/login', data = post_dict, cookies= firstget_cookie_dict ) # 點贊請求 dianzan_response = requests.post( url = 'http://dig.chouti.com/link/vote?linksId=11832246', cookies= firstget_cookie_dict ) print(dianzan_response.text) # 取消點贊 cancel_dianzan_response = requests.post( url = 'http://dig.chouti.com/vote/cancel/vote.do', cookies= firstget_cookie_dict, data={'linksId':11832246} ) print(cancel_dianzan_response.text) # 獲取個人信息 get_person_info_resonse = requests.get( url = 'http://dig.chouti.com/profile', cookies= firstget_cookie_dict, ) # 按照某種encoding方式編碼 get_person_info_resonse.encoding = get_person_info_resonse.apparent_encoding # 將其內容放入BS中進行解析 person_info_site = BeautifulSoup(get_person_info_resonse.text,features='html.parser') # 找到之后可以做任何處理，獲取配置中的nickname nickname_tag = person_info_site.find(id='nick') nickname = person_info_site.find(id='nick').attrs.get('value') print('昵稱：',nickname) # 更新自己在抽屜上的個人信息 personal_info = { 'jid': 'cdu_49017916793', 'nick': '努力哥', 'imgUrl': 'http://img2.chouti.com/CHOUTI_90A38B32473A49B7B26A49F46B34268C_W585H359=C60x60.png', # http://img2.chouti.com/CHOUTI_BAE7F736FE7B48E49D1CEE459020F3B0_W390H390=48x48.jpg 'sex': True, 'proveName': '北京', 'cityName': '澳門', 'sign': '黑hi呃呃哈發到付' } update_person_info_resonse = requests.post( url = 'http://dig.chouti.com/profile/update', cookies= firstget_cookie_dict, data=personal_info ) print(update_person_info_resonse.text) #########################Session方式登錄抽屜######################### session = requests.Session() # 先登陸一下抽屜網 i1 = session.get( url='http://dig.chouti.com/' ) # 模擬抽屜登錄 login_post_dict = { 'phone': '86188116*****', #86+手機號碼 'password': '******', #密碼 'oneMonth': 1 } i2 = session.post( url='http://dig.chouti.com/login', data=login_post_dict, )
3、自動登陸GitHub
#!/usr/bin/env python
# -*- coding: utf8 -*-
# date: 2017/5/10 16:32
import requests
from bs4 import BeautifulSoup
# GitHub是基於authenticity_token，具有預防csrf_token的功能
# 首先訪問頁面，獲取頁面上的authenticity_token
i1 = requests.get('https://github.com/login')
# print(i1.content)
login_page_res = BeautifulSoup(i1.content,features='lxml')
authenticity_token = login_page_res.find(name='input',attrs={'name':'authenticity_token'}).attrs.get('value')
cookies1 = i1.cookies.get_dict()
# print(authenticity_token)
form_data = {
'commit': 'Sign in',
'utf8': '✓',
'authenticity_token': authenticity_token,
'login': '*****',
'password': '******',
}
# 將數據封裝在post請求中進行登錄，而且要加上cookie
login_res = requests.post(
url='https://github.com/session',
data=form_data,
cookies=cookies1
)
# print(login_res.text)
# 拿到頁面中的自己的項目列表
login_page_res = BeautifulSoup(login_res.content,features='lxml')
list_info = login_page_res.select("span .repo")
for i in list_info:
print(i.text)
cookies1 = i1.cookies.get_dict()
4、自動登錄cnblog
博客園站用了一個rsa算法的加密模塊，所以安裝加密模塊。才能驗證登錄。
pip3 install rsa
代碼：
#!/usr/bin/env python
# -*- coding: utf8 -*-
# date: 2017/5/11 10:51
import re
import json
import base64
import rsa
import requests
from bs4 import BeautifulSoup
# 負責模仿前端js模塊對賬號和密碼加密
def js_enrypt(text):
# 先從博客園拿到public key
public_key = 'MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCp0wHYbg/NOPO3nzMD3dndwS0MccuMeXCHgVlGOoYyFwLdS24Im2e7YyhB0wrUsyYf0/nhzCzBK8ZC9eCWqd0aHbdgOQT6CuFQBMjbyGYvlVYU2ZP7kG9Ft6YV6oc9ambuO7nPZh+bvXH0zDKfi02prknrScAKC0XhadTHT3Al0QIDAQAB'
# 將拿到的一串字符，轉換成64進制
der = base64.standard_b64decode(public_key)
# 再將其轉換成數字，作為公鑰加載
pk = rsa.PublicKey.load_pkcs1_openssl_der(der)
# 運用公鑰對傳進來的文字進行加密
v1 = rsa.encrypt(bytes(text,'utf8'),pk)
# 對加密后的內容進行解碼
value = base64.encodebytes(v1).replace(b'\n',b'')
value = value.decode('utf8')
# 將其返回
return value
session = requests.Session()
# 寫個錯誤的用戶名和密碼，提交一下。就找到提交數據
post_data = {
'input1': js_enrypt('******'),
'input2': js_enrypt('******'),
'remember': True
}
# 發送一次請求，獲取ajax發送post時要發送的VerificationToken，需要將其放在請求頭部
login_page = session.get(
url='https://passport.cnblogs.com/user/signin',
)
VerificationToken = re.compile("'VerificationToken': '(.*)'")
v = re.search(VerificationToken,login_page.text)
VerificationToken = v.group(1)
# 發送請求，注意將數據json序列化，因為Accept:application/json
login_post_res = session.post(
url='https://passport.cnblogs.com/user/signin',
data=json.dumps(post_data),
headers={
'VerificationToken': VerificationToken,
'X-Requested-With': 'XMLHttpRequest',
'Content-Type': 'application/json; charset=UTF-8'
}
)
# 登錄賬號設置頁
setting_page = session.get(
url='https://home.cnblogs.com/set/account/',
)
soup = BeautifulSoup(setting_page.content,features='lxml')
name = soup.select_one('#loginName_display_block div').get_text().strip()
print('你的賬號名為:',name)
5、自動登錄知乎
#!/usr/bin/env python
# -*- coding: utf8 -*-
import requests
from bs4 import BeautifulSoup
session = requests.Session()
# 知乎會查看你的是否有用戶客戶端信息，沒有不會讓爬的
signin_page = session.get(
url='https://www.zhihu.com/#signin',
headers={
'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.98 Safari/537.36',
}
)
# 拿到頁面的_xrf為了防止csrf攻擊，post數據的時候需要提供
signin_page_tag = BeautifulSoup(signin_page.content,features='lxml')
xsrf_code = signin_page_tag.find('input',attrs={'name':'_xsrf'}).attrs.get('value')
# 從知乎服務器獲取驗證碼照片，發送請求POST，發現需要傳入以下三個參數
# r:1494416****
# type:login
# lang:cn
import time
current_time = time.time()
yanzhengma = session.get(
url='https://www.zhihu.com/captcha.gif',
params={
'r': current_time,
'type': 'login',
# 'lang': 'en' # 使用不同的語言，cn最為復雜，不加的話，最容易識別，en為立體的英文也不好識別
},
headers={
'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.98 Safari/537.36',
}
)
# 將從服務器收到的驗證碼寫入文件，可以查看啦
with open('zhihu.gif', 'wb') as f:
f.write(yanzhengma.content)
captcha = input("請打開照片查看驗證碼：")
form_data = {
'_xsrf': xsrf_code,
'password': '********',
'captcha': captcha,
# 'captcha': '{"img_size": [200, 44], "input_points": [[40.2, 34.2], [156.2, 28.2], [138.2, 24.2]]}',
# 'captcha_type': 'cn', # 如果為中文的驗證碼比較復雜
'phone_num': '***********', #填手機號碼登錄
# 'email':"sddasd@123.com" # 郵箱登錄的方式
}
login_response = session.post(
url='https://www.zhihu.com/login/phone_num', #前端會根據你的數據類型選擇用郵箱或者手機號碼登錄
# url='https://www.zhihu.com/login/phone_num'
data=form_data,
headers = {
'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.98 Safari/537.36',
}
)
index_page = session.get(
url='https://www.zhihu.com/',
headers={
'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.98 Safari/537.36',
}
)
index_page_tag = BeautifulSoup(index_page.content,features='lxml')
print(index_page_tag)
運行程序后，輸入驗證碼。登錄成功后，搜索用戶名稱，能找到我多個相同的用戶名稱，就說明登錄成功。
