近期在做爬蟲時有時會遇到網站只提供pdf的情況，這樣就不能使用scrapy直接抓取頁面內容了，只能通過解析PDF的方式處理，目前的解決方案大致只有pyPDF和PDFMiner。因為據說PDFMiner更適合文本的解析，而我需要解析的正是文本，因此最后選擇使用PDFMiner(這也就意味着我對pyPDF一無所知了)。
首先說明的是解析PDF是非常蛋疼的事，即使是PDFMiner對於格式不工整的PDF解析效果也不怎么樣，所以連PDFMiner的開發者都吐槽PDF is evil. 不過這些並不重要。官方文檔在此：http://www.unixuser.org/~euske/python/pdfminer/index.html
一.安裝：
1.首先下載源文件包 http://pypi.python.org/pypi/pdfminer/，解壓，然后命令行安裝即可：python setup.py install
2.安裝完成后使用該命令行測試：pdf2txt.py samples/simple1.pdf，如果顯示以下內容則表示安裝成功：
Hello World Hello World H e l l o W o r l d H e l l o W o r l d
3.如果要使用中日韓文字則需要先編譯再安裝：
# make cmap
python tools/conv_cmap.py pdfminer/cmap Adobe-CNS1 cmaprsrc/cid2code_Adobe_CNS1.txtreading 'cmaprsrc/cid2code_Adobe_CNS1.txt'...writing 'CNS1_H.py'......(this may take several minutes)
# python setup.py install
二．使用
由於解析PDF是一件非常耗時和內存的工作，因此PDFMiner使用了一種稱作lazy parsing的策略，只在需要的時候才去解析，以減少時間和內存的使用。要解析PDF至少需要兩個類：PDFParser 和 PDFDocument，PDFParser 從文件中提取數據，PDFDocument保存數據。另外還需要PDFPageInterpreter去處理頁面內容，PDFDevice將其轉換為我們所需要的。PDFResourceManager用於保存共享內容例如字體或圖片。
Figure 1. Relationships between PDFMiner classes
比較重要的是Layout，主要包括以下這些組件：
LTPage
Represents an entire page. May contain child objects like LTTextBox, LTFigure, LTImage, LTRect, LTCurve and LTLine.
LTTextBox
Represents a group of text chunks that can be contained in a rectangular area. Note that this box is created by geometric analysis and does not necessarily represents a logical boundary of the text. It contains a list of LTTextLine objects. get_text() method returns the text content.
LTTextLine
Contains a list of LTChar objects that represent a single text line. The characters are aligned either horizontaly or vertically, depending on the text's writing mode. get_text() method returns the text content.
LTChar
LTAnno
Represent an actual letter in the text as a Unicode string. Note that, while a LTChar object has actual boundaries, LTAnno objects does not, as these are "virtual" characters, inserted by a layout analyzer according to the relationship between two characters (e.g. a space).
LTFigure
Represents an area used by PDF Form objects. PDF Forms can be used to present figures or pictures by embedding yet another PDF document within a page. Note that LTFigure objects can appear recursively.
LTImage
Represents an image object. Embedded images can be in JPEG or other formats, but currently PDFMiner does not pay much attention to graphical objects.
LTLine
Represents a single straight line. Could be used for separating text or figures.
LTRect
Represents a rectangle. Could be used for framing another pictures or figures.
LTCurve
Represents a generic Bezier curve.
Figure 2. Layout objects and its tree structure
官方文檔給了幾個Demo但是都過於簡略，雖然給了一個詳細一些的Demo，但鏈接地址是舊的現在已經失效，不過最終還是找到了新的地址：http://denis.papathanasiou.org/posts/2010.08.04.post.html
這個Demo就比較詳細了，源碼如下：
1 #!/usr/bin/python 2 3 import sys 4 import os 5 from binascii import b2a_hex 6 7 8 ### 9 ### pdf-miner requirements 10 ### 11 12 from pdfminer.pdfparser import PDFParser 13 from pdfminer.pdfdocument import PDFDocument, PDFNoOutlines 14 from pdfminer.pdfpage import PDFPage 15 from pdfminer.pdfinterp import PDFResourceManager, PDFPageInterpreter 16 from pdfminer.converter import PDFPageAggregator 17 from pdfminer.layout import LAParams, LTTextBox, LTTextLine, LTFigure, LTImage, LTChar 18 19 def with_pdf (pdf_doc, fn, pdf_pwd, *args): 20 """Open the pdf document, and apply the function, returning the results""" 21 result = None 22 try: 23 # open the pdf file 24 fp = open(pdf_doc, 'rb') 25 # create a parser object associated with the file object 26 parser = PDFParser(fp) 27 # create a PDFDocument object that stores the document structure 28 doc = PDFDocument(parser, pdf_pwd) 29 # connect the parser and document objects 30 parser.set_document(doc) 31 # supply the password for initialization 32 33 if doc.is_extractable: 34 # apply the function and return the result 35 result = fn(doc, *args) 36 37 # close the pdf file 38 fp.close() 39 except IOError: 40 # the file doesn't exist or similar problem 41 pass 42 return result 43 44 45 ### 46 ### Table of Contents 47 ### 48 49 def _parse_toc (doc): 50 """With an open PDFDocument object, get the table of contents (toc) data 51 [this is a higher-order function to be passed to with_pdf()]""" 52 toc = [] 53 try: 54 outlines = doc.get_outlines() 55 for (level,title,dest,a,se) in outlines: 56 toc.append( (level, title) ) 57 except PDFNoOutlines: 58 pass 59 return toc 60 61 def get_toc (pdf_doc, pdf_pwd=''): 62 """Return the table of contents (toc), if any, for this pdf file""" 63 return with_pdf(pdf_doc, _parse_toc, pdf_pwd) 64 65 66 ### 67 ### Extracting Images 68 ### 69 70 def write_file (folder, filename, filedata, flags='w'): 71 """Write the file data to the folder and filename combination 72 (flags: 'w' for write text, 'wb' for write binary, use 'a' instead of 'w' for append)""" 73 result = False 74 if os.path.isdir(folder): 75 try: 76 file_obj = open(os.path.join(folder, filename), flags) 77 file_obj.write(filedata) 78 file_obj.close() 79 result = True 80 except IOError: 81 pass 82 return result 83 84 def determine_image_type (stream_first_4_bytes): 85 """Find out the image file type based on the magic number comparison of the first 4 (or 2) bytes""" 86 file_type = None 87 bytes_as_hex = b2a_hex(stream_first_4_bytes) 88 if bytes_as_hex.startswith('ffd8'): 89 file_type = '.jpeg' 90 elif bytes_as_hex == '89504e47': 91 file_type = '.png' 92 elif bytes_as_hex == '47494638': 93 file_type = '.gif' 94 elif bytes_as_hex.startswith('424d'): 95 file_type = '.bmp' 96 return file_type 97 98 def save_image (lt_image, page_number, images_folder): 99 """Try to save the image data from this LTImage object, and return the file name, if successful""" 100 result = None 101 if lt_image.stream: 102 file_stream = lt_image.stream.get_rawdata() 103 if file_stream: 104 file_ext = determine_image_type(file_stream[0:4]) 105 if file_ext: 106 file_name = ''.join([str(page_number), '_', lt_image.name, file_ext]) 107 if write_file(images_folder, file_name, file_stream, flags='wb'): 108 result = file_name 109 return result 110 111 112 ### 113 ### Extracting Text 114 ### 115 116 def to_bytestring (s, enc='utf-8'): 117 """Convert the given unicode string to a bytestring, using the standard encoding, 118 unless it's already a bytestring""" 119 if s: 120 if isinstance(s, str): 121 return s 122 else: 123 return s.encode(enc) 124 125 def update_page_text_hash (h, lt_obj, pct=0.2): 126 """Use the bbox x0,x1 values within pct% to produce lists of associated text within the hash""" 127 128 x0 = lt_obj.bbox[0] 129 x1 = lt_obj.bbox[2] 130 131 key_found = False 132 for k, v in h.items(): 133 hash_x0 = k[0] 134 if x0 >= (hash_x0 * (1.0-pct)) and (hash_x0 * (1.0+pct)) >= x0: 135 hash_x1 = k[1] 136 if x1 >= (hash_x1 * (1.0-pct)) and (hash_x1 * (1.0+pct)) >= x1: 137 # the text inside this LT* object was positioned at the same 138 # width as a prior series of text, so it belongs together 139 key_found = True 140 v.append(to_bytestring(lt_obj.get_text())) 141 h[k] = v 142 if not key_found: 143 # the text, based on width, is a new series, 144 # so it gets its own series (entry in the hash) 145 h[(x0,x1)] = [to_bytestring(lt_obj.get_text())] 146 147 return h 148 149 def parse_lt_objs (lt_objs, page_number, images_folder, text=[]): 150 """Iterate through the list of LT* objects and capture the text or image data contained in each""" 151 text_content = [] 152 153 page_text = {} # k=(x0, x1) of the bbox, v=list of text strings within that bbox width (physical column) 154 for lt_obj in lt_objs: 155 if isinstance(lt_obj, LTTextBox) or isinstance(lt_obj, LTTextLine): 156 # text, so arrange is logically based on its column width 157 page_text = update_page_text_hash(page_text, lt_obj) 158 elif isinstance(lt_obj, LTImage): 159 # an image, so save it to the designated folder, and note its place in the text 160 saved_file = save_image(lt_obj, page_number, images_folder) 161 if saved_file: 162 # use html style <img /> tag to mark the position of the image within the text 163 text_content.append('<img src="'+os.path.join(images_folder, saved_file)+'" />') 164 else: 165 print >> sys.stderr, "error saving image on page", page_number, lt_obj.__repr__ 166 elif isinstance(lt_obj, LTFigure): 167 # LTFigure objects are containers for other LT* objects, so recurse through the children 168 text_content.append(parse_lt_objs(lt_obj, page_number, images_folder, text_content)) 169 170 for k, v in sorted([(key,value) for (key,value) in page_text.items()]): 171 # sort the page_text hash by the keys (x0,x1 values of the bbox), 172 # which produces a top-down, left-to-right sequence of related columns 173 text_content.append(''.join(v)) 174 175 return '\n'.join(text_content) 176 177 178 ### 179 ### Processing Pages 180 ### 181 182 def _parse_pages (doc, images_folder): 183 """With an open PDFDocument object, get the pages and parse each one 184 [this is a higher-order function to be passed to with_pdf()]""" 185 rsrcmgr = PDFResourceManager() 186 laparams = LAParams() 187 device = PDFPageAggregator(rsrcmgr, laparams=laparams) 188 interpreter = PDFPageInterpreter(rsrcmgr, device) 189 190 text_content = [] 191 for i, page in enumerate(PDFPage.create_pages(doc)): 192 interpreter.process_page(page) 193 # receive the LTPage object for this page 194 layout = device.get_result() 195 # layout is an LTPage object which may contain child objects like LTTextBox, LTFigure, LTImage, etc. 196 text_content.append(parse_lt_objs(layout, (i+1), images_folder)) 197 198 return text_content 199 200 def get_pages (pdf_doc, pdf_pwd='', images_folder='/tmp'): 201 """Process each of the pages in this pdf file and return a list of strings representing the text found in each page""" 202 return with_pdf(pdf_doc, _parse_pages, pdf_pwd, *tuple([images_folder])) 203 204 a = open('a.txt','a') 205 for i in get_pages('/home/jamespei/nova.pdf'): 206 a.write(i) 207 a.close()
這段代碼重點在於第１２８行，可以看到PDFMiner是一種基於坐標來解析的框架，PDF中能解析的組件全都包括上下左右邊緣的坐標，如x0 = lt_obj.bbox[0]就是lt_obj元素的左邊緣的坐標，同理x1則為右邊緣。以上代碼的意思就是把所有x0且x1的坐標相差在２０％以內的元素分成一組，這樣就實現了從PDF文件中定向抽取內容。
－－－－－－－－－－－－－－－－－－－－－－－－－－－補充－－－－－－－－－－－－－－－－－－－－－－－－－
有一個需要注意的地方，在解析有些PDF的時候會報這樣的異常：pdfminer.pdfdocument.PDFEncryptionError: Unknown algorithm: param={'CF': {'StdCF': {'Length': 16, 'CFM': /AESV2, 'AuthEvent': /DocOpen}}, 'O': '\xe4\xe74\xb86/\xa8)\xa6x\xe6\xa3/U\xdf\x0fWR\x9cPh\xac\xae\x88B\x06_\xb0\x93@\x9f\x8d', 'Filter': /Standard, 'P': -1340, 'Length': 128, 'R': 4, 'U': '|UTX#f\xc9V\x18\x87z\x10\xcb\xf5{\xa7\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', 'V': 4, 'StmF': /StdCF, 'StrF': /StdCF}
從字面意思來看是因為這個PDF是一個加密的PDF，所以無法解析 ，但是如果直接打開PDF卻是可以的並沒有要求輸密碼什么的，原因是這個PDF雖然是加過密的，但密碼是空，所以就出現了這樣的問題。
解決這個的問題的辦法是通過qpdf命令來解密文件（要確保已經安裝了qpdf），要想在python中調用該命令只需使用call即可：
1 from subprocess import call 2 call('qpdf --password=%s --decrypt %s %s' %('', file_path, new_file_path), shell=True)
其中參數file_path是要解密的PDF的路徑，new_file_path是解密后的PDF文件路徑，然后使用解密后的文件去做解析就OK了