爬取煎蛋XXOO妹子圖片

今天回憶廖大的多線程的時候，看到下面有人寫了個多線程的爬蟲http://www.tendcode.com/article/jiandan-meizi-spider-2/，點進去看了下，分析的很仔細，寫了接近200行代碼吧

讓后我就研究了一下這個網站，emmmm，selenium + PhantomJS不就直接搞定了嘛，然后就寫了段code:

然后發現，哇，selenium不支持PhantomJS了，因為chrome和firefox自帶了headless的訪問，然后就去各個blog看，最后爬下了這個網站:

import unittest
import requests
import time
import re
from random import randint
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.keys import Keys

class ooxx_spider(unittest.TestCase):

    def setUp(self):
        chrome_options = Options()
        chrome_options.add_argument('--headless')
        chrome_options.add_argument('--disable-gpu')
        self.driver = webdriver.Chrome('E:/chromedriver.exe', chrome_options=chrome_options)

    def test_spider(self):
        for i in range(1, 80):
            url = 'http://jandan.net/ooxx/' + 'page-' + str(i)
            self.driver.get(url)
            print(url)
            elem = self.driver.find_elements_by_xpath('//*[@class="commentlist"]/li/div/div/div/p/img')#/li/div/div/div/p/img
            for j in elem:
                self.save_img(j.get_attribute('src'))
            print('第{}頁爬取成功'.format(i))

    def save_img(self, res):
        suffix = res.split('.')[-1]
        destination = 'picture/' + str(randint(1, 1000)) + str(randint(1, 1000)) + '.'+ suffix
        r = requests.get(res)
        with open(destination, 'wb') as f:
            f.write(r.content)

    def tearDown(self):
        self.driver.close()

if __name__ == '__main__':
    unittest.main()

補上多線程的代碼

核心代碼:

def test_multiscraping(self):
        p = Pool()#默認大小是cpu的核數，你可以修改比如說雙核Pool(2)
        #這里假設我是4個進程，所以range(5)
        for i in range(5):
            p.apply_async(scraping, args = (i, ))
        p.close()
        p.join()

cpu太垃圾了，晚上回去用同學的cpu測試一下(留下了窮人的眼淚)