scrapy和selenium結合抓取動態網頁


1、安裝python (我用的是2.7版本的)

2、安裝scrapy:   詳情請參考 http://blog.csdn.net/wukaibo1986/article/details/8167590 (提示,能下載源碼安裝的就避免用pip install **)

     安裝過程中遇到python擴展問題”unable to find vcvarsall.bat“的解決辦法: http://blog.csdn.net/ren911/article/details/6448696

3、安裝selenium,  https://pypi.python.org/pypi/selenium, 注意安裝后Download the server separately, from:http://selenium-release.storage.googleapis.com/2.42/selenium-server-standalone-2.42.0.jar,  然后運行(命令行運行,需要安裝Java Runtime Environment (JRE))  java -jar selenium-server-standalone-2.42.0.jar

4、建scrapy工程  , 參考 http://www.searchtb.com/2011/01/an-introduction-to-crawler.html?spm=0.0.0.0.HrlxVl

5、scrapy和selenium結合抓取動態網頁:  https://github.com/tmslav/google_adwords_scrape/blob/master/adwords_bot.py

7、Python Selenium socket error - [Errno 61] Connection refused:  http://stackoverflow.com/questions/12913991/python-selenium-socket-error-errno-61-connection-refused

8、Selenium: FirefoxProfile exception Can't load the profile: https://github.com/tmslav/google_adwords_scrape/blob/master/adwords_bot.py


免責聲明!

本站轉載的文章為個人學習借鑒使用,本站對版權不負任何法律責任。如果侵犯了您的隱私權益,請聯系本站郵箱yoyou2525@163.com刪除。



 
粵ICP備18138465號   © 2018-2025 CODEPRJ.COM