python里如何計算大文件的md5

在python3中，有了一個hashlib，可以用來計算md5，這里先給出一個簡單的例子：

import hashlib

sstr="i love hanyu"
print(hashlib.md5(sstr).hexdigest())

很遺憾的，出錯了，錯誤信息是：

C:\Python35\python.exe C:/pylearn/bottlelearn/3.py
Traceback (most recent call last):
  File "C:/pylearn/bottlelearn/3.py", line 4, in <module>
    print(hashlib.md5(sstr).hexdigest())
TypeError: Unicode-objects must be encoded before hashing

Process finished with exit code 1

這里主要是考慮到傳入的編碼不同，會導致md5出問題，所以，要求傳入前進行統一的編碼，修改如下：

 import hashlib
 hashlib.sha256(str(random.getrandbits(256)).encode('utf-8')).hexdigest()

import hashlib

with open(hash_file) as file:
    control_hash = file.readline().rstrip("\n")

wordlistfile = open(wordlist, "rb")
# ...
for line in wordlistfile:
    if hashlib.md5(line.rstrip(b'\n\r')).hexdigest() == control_hash:

 

下面，來看看如何計算大文件的md5，如果只是簡單的把文件都入到內存中，大文件會導致出現大問題，編碼如下：

import hashlib

def hash_bytestr_iter(bytesiter, hasher, ashexstr=False):
    for block in bytesiter:
        hasher.update(block)
    return (hasher.hexdigest() if ashexstr else hasher.digest())

def file_as_blockiter(afile, blocksize=65536):
    with afile:
        block = afile.read(blocksize)
        while len(block) > 0:
            yield block
            block = afile.read(blocksize)


[(fname, hash_bytestr_iter(file_as_blockiter(open(fname, 'rb')), hashlib.md5()))
    for fname in fnamelst]