python里如何计算大文件的md5

在python3中，有了一个hashlib，可以用来计算md5，这里先给出一个简单的例子：

import hashlib

sstr="i love hanyu"
print(hashlib.md5(sstr).hexdigest())

很遗憾的，出错了，错误信息是：

C:\Python35\python.exe C:/pylearn/bottlelearn/3.py
Traceback (most recent call last):
  File "C:/pylearn/bottlelearn/3.py", line 4, in <module>
    print(hashlib.md5(sstr).hexdigest())
TypeError: Unicode-objects must be encoded before hashing

Process finished with exit code 1

这里主要是考虑到传入的编码不同，会导致md5出问题，所以，要求传入前进行统一的编码，修改如下：

 import hashlib
 hashlib.sha256(str(random.getrandbits(256)).encode('utf-8')).hexdigest()

import hashlib

with open(hash_file) as file:
    control_hash = file.readline().rstrip("\n")

wordlistfile = open(wordlist, "rb")
# ...
for line in wordlistfile:
    if hashlib.md5(line.rstrip(b'\n\r')).hexdigest() == control_hash:

 

下面，来看看如何计算大文件的md5，如果只是简单的把文件都入到内存中，大文件会导致出现大问题，编码如下：

import hashlib

def hash_bytestr_iter(bytesiter, hasher, ashexstr=False):
    for block in bytesiter:
        hasher.update(block)
    return (hasher.hexdigest() if ashexstr else hasher.digest())

def file_as_blockiter(afile, blocksize=65536):
    with afile:
        block = afile.read(blocksize)
        while len(block) > 0:
            yield block
            block = afile.read(blocksize)


[(fname, hash_bytestr_iter(file_as_blockiter(open(fname, 'rb')), hashlib.md5()))
    for fname in fnamelst]