python之https爬蟲出現 SSL: CERTIFICATE_VERIFY_FAILED (同時打開fiddler就會出現)

1.參考

Python 升級到 2.7.9 之后引入了一個新特性，當你urllib.urlopen一個 https 的時候，會驗證一次 SSL 證書，當目標網站使用的是自簽名的證書時就會爆出一個 urllib2.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)> 的錯誤消息

The problem you are having is caused by an untrusted SSL certificate.

Like @dirk mentioned in a previous comment, the quickest fix is setting verify=False.

Please note that this will cause the certificate not to be verified. This will expose your application to security risks, such as man-in-the-middle attacks.

Of course, apply judgment. As mentioned in the comments, this may be acceptable for quick/throwaway applications/scripts, but really should not go to production software.

If just skipping the certificate check is not acceptable in your particular context, consider the following options, your best option is to set the verify parameter to a string that is the path of the .pem file of the certificate (which you should obtain by some sort of secure means).

So, as of version 2.0, the verify parameter accepts the following values, with their respective semantics:

True: causes the certificate to validated against the library's own trusted certificate authorities (Note: you can see which Root Certificates Requests uses via the Certifi library, a trust database of RCs extracted from Requests: Certifi - Trust Database for Humans).
False: bypasses certificate validation completely.
Path to a CA_BUNDLE file for Requests to use to validate the certificates.

Source: Requests - SSL Cert Verification

Also take a look at the cert parameter on the same link.

2.解決辦法

(1)全局設置，對requests不生效

ssl._create_default_https_context = ssl._create_unverified_context

(2)urllib2.urlopen 傳參

context = ssl._create_unverified_context()

urllib2.urlopen(context=context)

(3)參考 urllib2.urlopen 為opener疊加handler

# C:\Program Files\Anaconda2\Lib\urllib2.py
    # def urlopen
        # elif context:
            # https_handler = HTTPSHandler(context=context)
            # opener = build_opener(https_handler)   

# urllib2.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)>
try:
    opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cookiejar))
    resp = opener.open(req)        
except urllib2.URLError as err:
    print err
    opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cookiejar), urllib2.HTTPSHandler(context=context))  #疊加多個 handler
    resp = opener.open(req)

(4)request.get 傳參

# requests.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)
try:
    r = requests.get(url, headers=headers, cookies=cookie)
except requests.exceptions.SSLError as err:
    print err
    r = requests.get(url, headers=headers, cookies=cookie, verify=False)