【寫在開頭】
參考文章后面給出了備注信息,是在解決這個問題的時候,查找的比較有亮點的參考文章,如果本文章寫的不太清楚的,可以去原文章進行查看。下面列舉的四個文章有參考的成分也有驗證的成分,解決辦法重點參考的是文章4,前面3個只是對pycharm使用和安裝包的便捷性上比較有亮點,這里還是進行引用。后面方便對參考文章進行引用,對文獻進行編號。
【參考文章】:
1、https://www.cnblogs.com/zengsf/p/9801966.html
pycharm的設置選項中安裝文件,只是作為了解,知道可以通過pycharm平台安裝和卸載包,該文章也介紹了不能直接安裝twisted,所以還是需要下載安裝,我是直接從文獻4中給的百度盤下載安裝的,twisted安裝成功之后,可以成功安裝scrapy包,但是不能查看scrapy版本,提示scrapy不是內部命令,放棄了,直接看正文部分教程或者文章4。
2、http://www.manongjc.com/article/44529.html
提到的用豆瓣源進行加速安裝的命令,對了一下,直接pip install scrapy和使用豆瓣源安裝,后者的確很快,看了下https://pypi.douban.com/simple 源社區里面很多包,安裝比較慢的時候可以考慮使用,作為了解。
3、https://www.cnblogs.com/llssx/p/8378832.html
給出安裝scrapy時會遇到的一些錯誤提示,和安裝scrapy時自下而上的順序結構圖,只是作為了解,文章沒有給出解決方案。
4、https://blog.csdn.net/YiXiao1997/article/details/96345737
是一個完整的安裝流程,並且給出Twisted的百度盤文件,很方便。文章最后對該教程與該文章進行了對比。
【正文部分】
1、由於入手python的時候就用的pycharm,所以學到scrapy的時候就想着在pycharm下進行一些配置,pycharm有個控制台,見圖1,可以直接輸入命令行進行安裝操作,本來以為直接pip install scrapy就可以了,但事實並不是這樣,下圖2給出了提示的錯誤信息,實際提示的錯誤信息有將近一千多行,直接去最后比較重要的兩行,大概意思就是:安裝twisted出現錯誤,Microsoft Visual C++ 14.0文件缺失。(備注:錯誤提示內容在文獻3提到了)

圖1 pycharm控制台

圖2 錯誤提示信息:安裝twisted出現錯誤,Microsoft Visual C++ 14.0文件缺失
2、接着根據文獻4的提示,下載好twisted的安裝包,在pycharm的控制台進行安裝程序,安裝提示信息如下所示:
D:\zhuomian>pip install Twisted-19.2.0-cp37-cp37m-win_amd64.whl Processing d:\zhuomian\twisted-19.2.0-cp37-cp37m-win_amd64.whl Requirement already satisfied: constantly>=15.1 in c:\users\余生,別無他求\appdata\local\programs\python\python37\lib\site-packages (from Twiste d==19.2.0) (15.1.0) Requirement already satisfied: attrs>=17.4.0 in c:\users\余生,別無他求\appdata\local\programs\python\python37\lib\site-packages (from Twisted== 19.2.0) (19.1.0) Requirement already satisfied: PyHamcrest>=1.9.0 in c:\users\余生,別無他求\appdata\local\programs\python\python37\lib\site-packages (from Twist ed==19.2.0) (1.9.0) Requirement already satisfied: zope.interface>=4.4.2 in c:\users\余生,別無他求\appdata\local\programs\python\python37\lib\site-packages (from T wisted==19.2.0) (4.6.0) Requirement already satisfied: incremental>=16.10.1 in c:\users\余生,別無他求\appdata\local\programs\python\python37\lib\site-packages (from Tw isted==19.2.0) (17.5.0) Requirement already satisfied: Automat>=0.3.0 in c:\users\余生,別無他求\appdata\local\programs\python\python37\lib\site-packages (from Twisted= =19.2.0) (0.7.0) Requirement already satisfied: hyperlink>=17.1.1 in c:\users\余生,別無他求\appdata\local\programs\python\python37\lib\site-packages (from Twist ed==19.2.0) (19.0.0) Requirement already satisfied: six in c:\users\余生,別無他求\appdata\local\programs\python\python37\lib\site-packages (from PyHamcrest>=1.9.0-> Twisted==19.2.0) (1.12.0) Requirement already satisfied: setuptools in c:\users\余生,別無他求\appdata\local\programs\python\python37\lib\site-packages (from PyHamcrest>= 1.9.0->Twisted==19.2.0) (39.0.1) Requirement already satisfied: idna>=2.5 in c:\users\余生,別無他求\appdata\local\programs\python\python37\lib\site-packages (from hyperlink>=17 .1.1->Twisted==19.2.0) (2.8) Installing collected packages: Twisted Successfully installed Twisted-19.2.0
3、安裝好twisted之后,使用命令pip install scrapy繼續安裝scrapy,下面是控制台提示信息,成功安裝。
D:\zhuomian>pip install scrapy Collecting scrapy Using cached https://files.pythonhosted.org/packages/a3/b1/d1ab5b3f84640097cf5ff642e2e357546781746d4fec2ebb40432904c57d/Scrapy-1.7.2-py2.py3-n one-any.whl Requirement already satisfied: lxml in c:\users\余生,別無他求\appdata\roaming\python\python37\site-packages (from scrapy) (4.3.4) Requirement already satisfied: pyOpenSSL in c:\users\余生,別無他求\appdata\roaming\python\python37\site-packages (from scrapy) (19.0.0) Requirement already satisfied: Twisted>=13.1.0; python_version != "3.4" in c:\users\余生,別無他求\appdata\local\programs\python\python37\lib\si te-packages (from scrapy) (19.2.0) Requirement already satisfied: queuelib in c:\users\余生,別無他求\appdata\roaming\python\python37\site-packages (from scrapy) (1.5.0) Requirement already satisfied: six>=1.5.2 in c:\users\余生,別無他求\appdata\local\programs\python\python37\lib\site-packages (from scrapy) (1.1 2.0) Requirement already satisfied: service-identity in c:\users\余生,別無他求\appdata\local\programs\python\python37\lib\site-packages (from scrapy ) (18.1.0) Requirement already satisfied: cssselect>=0.9 in c:\users\余生,別無他求\appdata\roaming\python\python37\site-packages (from scrapy) (1.0.3) Requirement already satisfied: w3lib>=1.17.0 in c:\users\余生,別無他求\appdata\roaming\python\python37\site-packages (from scrapy) (1.20.0) Requirement already satisfied: PyDispatcher>=2.0.5 in c:\users\余生,別無他求\appdata\local\programs\python\python37\lib\site-packages (from scr apy) (2.0.5) Requirement already satisfied: parsel>=1.5 in c:\users\余生,別無他求\appdata\roaming\python\python37\site-packages (from scrapy) (1.5.1) Requirement already satisfied: cryptography>=2.3 in c:\users\余生,別無他求\appdata\local\programs\python\python37\lib\site-packages (from pyOpe nSSL->scrapy) (2.7) Requirement already satisfied: PyHamcrest>=1.9.0 in c:\users\余生,別無他求\appdata\local\programs\python\python37\lib\site-packages (from Twist ed>=13.1.0; python_version != "3.4"->scrapy) (1.9.0) Requirement already satisfied: incremental>=16.10.1 in c:\users\余生,別無他求\appdata\local\programs\python\python37\lib\site-packages (from Tw isted>=13.1.0; python_version != "3.4"->scrapy) (17.5.0) Requirement already satisfied: hyperlink>=17.1.1 in c:\users\余生,別無他求\appdata\local\programs\python\python37\lib\site-packages (from Twist ed>=13.1.0; python_version != "3.4"->scrapy) (19.0.0) Requirement already satisfied: Automat>=0.3.0 in c:\users\余生,別無他求\appdata\local\programs\python\python37\lib\site-packages (from Twisted> =13.1.0; python_version != "3.4"->scrapy) (0.7.0) Requirement already satisfied: constantly>=15.1 in c:\users\余生,別無他求\appdata\local\programs\python\python37\lib\site-packages (from Twiste d>=13.1.0; python_version != "3.4"->scrapy) (15.1.0) Requirement already satisfied: zope.interface>=4.4.2 in c:\users\余生,別無他求\appdata\local\programs\python\python37\lib\site-packages (from T wisted>=13.1.0; python_version != "3.4"->scrapy) (4.6.0) Requirement already satisfied: attrs>=17.4.0 in c:\users\余生,別無他求\appdata\local\programs\python\python37\lib\site-packages (from Twisted>= 13.1.0; python_version != "3.4"->scrapy) (19.1.0) Requirement already satisfied: pyasn1 in c:\users\余生,別無他求\appdata\local\programs\python\python37\lib\site-packages (from service-identity ->scrapy) (0.4.5) Requirement already satisfied: pyasn1-modules in c:\users\余生,別無他求\appdata\local\programs\python\python37\lib\site-packages (from service- identity->scrapy) (0.2.5) Requirement already satisfied: cffi!=1.11.3,>=1.8 in c:\users\余生,別無他求\appdata\local\programs\python\python37\lib\site-packages (from cryp tography>=2.3->pyOpenSSL->scrapy) (1.12.3) Requirement already satisfied: asn1crypto>=0.21.0 in c:\users\余生,別無他求\appdata\local\programs\python\python37\lib\site-packages (from cryp tography>=2.3->pyOpenSSL->scrapy) (0.24.0) Requirement already satisfied: setuptools in c:\users\余生,別無他求\appdata\local\programs\python\python37\lib\site-packages (from PyHamcrest>= 1.9.0->Twisted>=13.1.0; python_version != "3.4"->scrapy) (39.0.1) Requirement already satisfied: idna>=2.5 in c:\users\余生,別無他求\appdata\local\programs\python\python37\lib\site-packages (from hyperlink>=17 .1.1->Twisted>=13.1.0; python_version != "3.4"->scrapy) (2.8) Requirement already satisfied: pycparser in c:\users\余生,別無他求\appdata\local\programs\python\python37\lib\site-packages (from cffi!=1.11.3, >=1.8->cryptography>=2.3->pyOpenSSL->scrapy) (2.19) Installing collected packages: scrapy Successfully installed scrapy-1.7.2
4、進行驗證,查看scrapy版本,沒有提示scrapy不是內部命令,也給出了版本信息,說明安裝成功
D:\zhuomian>scrapy Scrapy 1.7.2 - no active project Usage: scrapy <command> [options] [args] Available commands: bench Run quick benchmark test fetch Fetch a URL using the Scrapy downloader genspider Generate new spider using pre-defined templates runspider Run a self-contained spider (without creating a project) settings Get settings values shell Interactive scraping console startproject Create new project version Print Scrapy version view Open URL in browser, as seen by Scrapy [ more ] More commands available when run from project directory Use "scrapy <command> -h" to see more info about a command
【寫在最后】
就像前面說的,這篇文章是自己對在學習scrapy環境配置的時候,遇到的問題的一個記錄,參考的還是網上的解決辦法,但同時也是對網上的教程的一個驗證,前面四篇文章的備注都作出了說明,對比一下第四篇參考文章,我這里沒有安裝wheel工具包,直接安裝的twisted,也是可以成功安裝的。
寫到這里,只是進行了環境配置,還沒有進行驗證,是不是可以成功進行項目開發,下一篇進行講解,scrapy簡單的項目開發。
小編尊重原創,文章如有侵權,聯系小編刪帖,謝謝監督。轉發請備注出處!
