序列化
Python中用於序列化的兩個模塊
- json 用於【字符串】和 【python基本數據類型】 間進行轉換
- pickle 用於【python特有的類型】 和 【python基本數據類型】間進行轉換
Json模塊提供了四個功能:dumps、dump、loads、load
pickle模塊提供了四個功能:dumps、dump、loads、load
json模塊
# json()將字符串形式的列表或字典轉換為list或dict類型,json是所有語言相互通信的方式
# 注意外層字符形式一定是''單引號,'{"a":"xiao","b":"xiao"}'列表或字典中的字符串一定要""雙引號,否則報錯
import json # json.dumps() 將python基本數據類型轉化成字符串形式 dic = {'k1':'v1'} print(dic,type(dic)) result = json.dumps(dic) print(result,type(result)) # json.loads() 將python字符串形式轉化成基本數據類型 s = '{"k1":123}' dic = json.loads(s) print(dic,type(dic)) # json.dump() 先序列化,再寫入文件 li = [11,22,33] json.dump(li,open('db','w')) # json.load() 讀取文件反序列化 l = json.load(open('db','r')) print(l,type(l))
pickle模塊
pickple只有python才能用,用於復雜類型的序列化,(如果是序列化一個對象,在別的模塊中反序列化的時候一定要導入該對象所屬的類,否則報錯)
import pickle # pickle.dumps() 序列化 li = [11,22,33] r = pickle.dumps(li) print(r) # pickle.loads() 反序列化 result = pickle.loads(r) print(result,type(result)) # pickle.dump() 先序列化,再寫入文件 l1 = [11,22,33,55] pickle.dump(l1,open('db','wb')) # pickle.load() 讀取文件反序列化 result1 = pickle.load(open('db','rb')) print(result1,type(result1))
XML模塊
xml是實現不同語言或程序之間進行數據交換的協議,跟json差不多,但json使用起來更簡單,不過,古時候,在json還沒誕生的黑暗年代,大家只能選擇用xml呀,至今很多傳統公司如金融行業的很多系統的接口還主要是xml,XML文件格式如下:
<data> <country name="Liechtenstein"> <rank updated="yes">2</rank> <year>2023</year> <gdppc>141100</gdppc> <neighbor direction="E" name="Austria" /> <neighbor direction="W" name="Switzerland" /> </country> <country name="Singapore"> <rank updated="yes">5</rank> <year>2026</year> <gdppc>59900</gdppc> <neighbor direction="N" name="Malaysia" /> </country> <country name="Panama"> <rank updated="yes">69</rank> <year>2026</year> <gdppc>13600</gdppc> <neighbor direction="W" name="Costa Rica" /> <neighbor direction="E" name="Colombia" /> </country> </data>
1、解析XML兩種方法
(1)利用ElementTree.XML將字符串解析成xml對象
from xml.etree import ElementTree as ET # 打開文件,讀取XML內容 str_xml = open('xo.xml', 'r').read() # 將字符串解析成xml特殊對象,root代指xml文件的根節點 root = ET.XML(str_xml)
(2)利用ElementTree.parse將文件直接解析成xml對象
from xml.etree import ElementTree as ET # 直接解析xml文件 tree = ET.parse("xo.xml") # 獲取xml文件的根節點 root = tree.getroot()
2、操作XML
XML格式類型是節點嵌套節點,對於每一個節點均有以下功能,以便對當前節點進行操作:

class Element: """An XML element. This class is the reference implementation of the Element interface. An element's length is its number of subelements. That means if you want to check if an element is truly empty, you should check BOTH its length AND its text attribute. The element tag, attribute names, and attribute values can be either bytes or strings. *tag* is the element name. *attrib* is an optional dictionary containing element attributes. *extra* are additional element attributes given as keyword arguments. Example form: <tag attrib>text<child/>...</tag>tail """ 當前節點的標簽名 tag = None """The element's name.""" 當前節點的屬性 attrib = None """Dictionary of the element's attributes.""" 當前節點的內容 text = None """ Text before first subelement. This is either a string or the value None. Note that if there is no text, this attribute may be either None or the empty string, depending on the parser. """ tail = None """ Text after this element's end tag, but before the next sibling element's start tag. This is either a string or the value None. Note that if there was no text, this attribute may be either None or an empty string, depending on the parser. """ def __init__(self, tag, attrib={}, **extra): if not isinstance(attrib, dict): raise TypeError("attrib must be dict, not %s" % ( attrib.__class__.__name__,)) attrib = attrib.copy() attrib.update(extra) self.tag = tag self.attrib = attrib self._children = [] def __repr__(self): return "<%s %r at %#x>" % (self.__class__.__name__, self.tag, id(self)) def makeelement(self, tag, attrib): 創建一個新節點 """Create a new element with the same type. *tag* is a string containing the element name. *attrib* is a dictionary containing the element attributes. Do not call this method, use the SubElement factory function instead. """ return self.__class__(tag, attrib) def copy(self): """Return copy of current element. This creates a shallow copy. Subelements will be shared with the original tree. """ elem = self.makeelement(self.tag, self.attrib) elem.text = self.text elem.tail = self.tail elem[:] = self return elem def __len__(self): return len(self._children) def __bool__(self): warnings.warn( "The behavior of this method will change in future versions. " "Use specific 'len(elem)' or 'elem is not None' test instead.", FutureWarning, stacklevel=2 ) return len(self._children) != 0 # emulate old behaviour, for now def __getitem__(self, index): return self._children[index] def __setitem__(self, index, element): # if isinstance(index, slice): # for elt in element: # assert iselement(elt) # else: # assert iselement(element) self._children[index] = element def __delitem__(self, index): del self._children[index] def append(self, subelement): 為當前節點追加一個子節點 """Add *subelement* to the end of this element. The new element will appear in document order after the last existing subelement (or directly after the text, if it's the first subelement), but before the end tag for this element. """ self._assert_is_element(subelement) self._children.append(subelement) def extend(self, elements): 為當前節點擴展 n 個子節點 """Append subelements from a sequence. *elements* is a sequence with zero or more elements. """ for element in elements: self._assert_is_element(element) self._children.extend(elements) def insert(self, index, subelement): 在當前節點的子節點中插入某個節點,即:為當前節點創建子節點,然后插入指定位置 """Insert *subelement* at position *index*.""" self._assert_is_element(subelement) self._children.insert(index, subelement) def _assert_is_element(self, e): # Need to refer to the actual Python implementation, not the # shadowing C implementation. if not isinstance(e, _Element_Py): raise TypeError('expected an Element, not %s' % type(e).__name__) def remove(self, subelement): 在當前節點在子節點中刪除某個節點 """Remove matching subelement. Unlike the find methods, this method compares elements based on identity, NOT ON tag value or contents. To remove subelements by other means, the easiest way is to use a list comprehension to select what elements to keep, and then use slice assignment to update the parent element. ValueError is raised if a matching element could not be found. """ # assert iselement(element) self._children.remove(subelement) def getchildren(self): 獲取所有的子節點(廢棄) """(Deprecated) Return all subelements. Elements are returned in document order. """ warnings.warn( "This method will be removed in future versions. " "Use 'list(elem)' or iteration over elem instead.", DeprecationWarning, stacklevel=2 ) return self._children def find(self, path, namespaces=None): 獲取第一個尋找到的子節點 """Find first matching element by tag name or path. *path* is a string having either an element tag or an XPath, *namespaces* is an optional mapping from namespace prefix to full name. Return the first matching element, or None if no element was found. """ return ElementPath.find(self, path, namespaces) def findtext(self, path, default=None, namespaces=None): 獲取第一個尋找到的子節點的內容 """Find text for first matching element by tag name or path. *path* is a string having either an element tag or an XPath, *default* is the value to return if the element was not found, *namespaces* is an optional mapping from namespace prefix to full name. Return text content of first matching element, or default value if none was found. Note that if an element is found having no text content, the empty string is returned. """ return ElementPath.findtext(self, path, default, namespaces) def findall(self, path, namespaces=None): 獲取所有的子節點 """Find all matching subelements by tag name or path. *path* is a string having either an element tag or an XPath, *namespaces* is an optional mapping from namespace prefix to full name. Returns list containing all matching elements in document order. """ return ElementPath.findall(self, path, namespaces) def iterfind(self, path, namespaces=None): 獲取所有指定的節點,並創建一個迭代器(可以被for循環) """Find all matching subelements by tag name or path. *path* is a string having either an element tag or an XPath, *namespaces* is an optional mapping from namespace prefix to full name. Return an iterable yielding all matching elements in document order. """ return ElementPath.iterfind(self, path, namespaces) def clear(self): 清空節點 """Reset element. This function removes all subelements, clears all attributes, and sets the text and tail attributes to None. """ self.attrib.clear() self._children = [] self.text = self.tail = None def get(self, key, default=None): 獲取當前節點的屬性值 """Get element attribute. Equivalent to attrib.get, but some implementations may handle this a bit more efficiently. *key* is what attribute to look for, and *default* is what to return if the attribute was not found. Returns a string containing the attribute value, or the default if attribute was not found. """ return self.attrib.get(key, default) def set(self, key, value): 為當前節點設置屬性值 """Set element attribute. Equivalent to attrib[key] = value, but some implementations may handle this a bit more efficiently. *key* is what attribute to set, and *value* is the attribute value to set it to. """ self.attrib[key] = value def keys(self): 獲取當前節點的所有屬性的 key """Get list of attribute names. Names are returned in an arbitrary order, just like an ordinary Python dict. Equivalent to attrib.keys() """ return self.attrib.keys() def items(self): 獲取當前節點的所有屬性值,每個屬性都是一個鍵值對 """Get element attributes as a sequence. The attributes are returned in arbitrary order. Equivalent to attrib.items(). Return a list of (name, value) tuples. """ return self.attrib.items() def iter(self, tag=None): 在當前節點的子孫中根據節點名稱尋找所有指定的節點,並返回一個迭代器(可以被for循環)。 """Create tree iterator. The iterator loops over the element and all subelements in document order, returning all elements with a matching tag. If the tree structure is modified during iteration, new or removed elements may or may not be included. To get a stable set, use the list() function on the iterator, and loop over the resulting list. *tag* is what tags to look for (default is to return all elements) Return an iterator containing all the matching elements. """ if tag == "*": tag = None if tag is None or self.tag == tag: yield self for e in self._children: yield from e.iter(tag) # compatibility def getiterator(self, tag=None): # Change for a DeprecationWarning in 1.4 warnings.warn( "This method will be removed in future versions. " "Use 'elem.iter()' or 'list(elem.iter())' instead.", PendingDeprecationWarning, stacklevel=2 ) return list(self.iter(tag)) def itertext(self): 在當前節點的子孫中根據節點名稱尋找所有指定的節點的內容,並返回一個迭代器(可以被for循環)。 """Create text iterator. The iterator loops over the element and all subelements in document order, returning all inner text. """ tag = self.tag if not isinstance(tag, str) and tag is not None: return if self.text: yield self.text for e in self: yield from e.itertext() if e.tail: yield e.tail
由於 每個節點 都具有以上的方法,並且在上一步驟中解析時均得到了root(xml文件的根節點),so 可以利用以上方法進行操作xml文件。
(1)遍歷XML文檔的所有內容
from xml.etree import ElementTree as ET ############ 解析方式一 ############ """ # 打開文件,讀取XML內容 str_xml = open('xo.xml', 'r').read() # 將字符串解析成xml特殊對象,root代指xml文件的根節點 root = ET.XML(str_xml) """ ############ 解析方式二 ############ # 直接解析xml文件 tree = ET.parse("xo.xml") # 獲取xml文件的根節點 root = tree.getroot() ### 操作 # 頂層標簽 print(root.tag) # 遍歷XML文檔的第二層 for child in root: # 第二層節點的標簽名稱和標簽屬性 print(child.tag, child.attrib) # 遍歷XML文檔的第三層 for i in child: # 第二層節點的標簽名稱和內容 print(i.tag,i.text)
(2)遍歷XML中指定的節點
from xml.etree import ElementTree as ET ############ 解析方式一 ############ """ # 打開文件,讀取XML內容 str_xml = open('xo.xml', 'r').read() # 將字符串解析成xml特殊對象,root代指xml文件的根節點 root = ET.XML(str_xml) """ ############ 解析方式二 ############ # 直接解析xml文件 tree = ET.parse("xo.xml") # 獲取xml文件的根節點 root = tree.getroot() ### 操作 # 頂層標簽 print(root.tag) # 遍歷XML中所有的year節點 for node in root.iter('year'): # 節點的標簽名稱和內容 print(node.tag, node.text)
(3)修改節點內容
由於修改的節點時,均是在內存中進行,其不會影響文件中的內容。所以,如果想要修改,則需要重新將內存中的內容寫到文件。
解析字符串方式,修改,保存
from xml.etree import ElementTree as ET ############ 解析方式一 ############ # 打開文件,讀取XML內容 str_xml = open('xo.xml', 'r').read() # 將字符串解析成xml特殊對象,root代指xml文件的根節點 root = ET.XML(str_xml) ############ 操作 ############ # 頂層標簽 print(root.tag) # 循環所有的year節點 for node in root.iter('year'): # 將year節點中的內容自增一 new_year = int(node.text) + 1 node.text = str(new_year) # 設置屬性 node.set('name', 'alex') node.set('age', '18') # 刪除屬性 del node.attrib['name'] ############ 保存文件 ############ tree = ET.ElementTree(root) tree.write("newnew.xml", encoding='utf-8')
解析文件方式,修改,保存
from xml.etree import ElementTree as ET ############ 解析方式二 ############ # 直接解析xml文件 tree = ET.parse("xo.xml") # 獲取xml文件的根節點 root = tree.getroot() ############ 操作 ############ # 頂層標簽 print(root.tag) # 循環所有的year節點 for node in root.iter('year'): # 將year節點中的內容自增一 new_year = int(node.text) + 1 node.text = str(new_year) # 設置屬性 node.set('name', 'alex') node.set('age', '18') # 刪除屬性 del node.attrib['name'] ############ 保存文件 ############ tree.write("newnew.xml", encoding='utf-8') 解析文件方式,修改,保存
(4)刪除節點
解析字符串方式打開,刪除,保存
from xml.etree import ElementTree as ET ############ 解析字符串方式打開 ############ # 打開文件,讀取XML內容 str_xml = open('xo.xml', 'r').read() # 將字符串解析成xml特殊對象,root代指xml文件的根節點 root = ET.XML(str_xml) ############ 操作 ############ # 頂層標簽 print(root.tag) # 遍歷data下的所有country節點 for country in root.findall('country'): # 獲取每一個country節點下rank節點的內容 rank = int(country.find('rank').text) if rank > 50: # 刪除指定country節點 root.remove(country) ############ 保存文件 ############ tree = ET.ElementTree(root) tree.write("newnew.xml", encoding='utf-8')
解析文件方式打開,刪除,保存
from xml.etree import ElementTree as ET ############ 解析文件方式 ############ # 直接解析xml文件 tree = ET.parse("xo.xml") # 獲取xml文件的根節點 root = tree.getroot() ############ 操作 ############ # 頂層標簽 print(root.tag) # 遍歷data下的所有country節點 for country in root.findall('country'): # 獲取每一個country節點下rank節點的內容 rank = int(country.find('rank').text) if rank > 50: # 刪除指定country節點 root.remove(country) ############ 保存文件 ############ tree.write("newnew.xml", encoding='utf-8')
3、創建XML文檔
創建方式(一)
from xml.etree import ElementTree as ET # 創建根節點 root = ET.Element("famliy") # 創建節點大兒子 son1 = ET.Element('son', {'name': '兒1'}) # 創建小兒子 son2 = ET.Element('son', {"name": '兒2'}) # 在大兒子中創建兩個孫子 grandson1 = ET.Element('grandson', {'name': '兒11'}) grandson2 = ET.Element('grandson', {'name': '兒12'}) son1.append(grandson1) son1.append(grandson2) # 把兒子添加到根節點中 root.append(son1) root.append(son1) tree = ET.ElementTree(root) tree.write('oooo.xml',encoding='utf-8', short_empty_elements=False)
創建方式(一)
from xml.etree import ElementTree as ET # 創建根節點 root = ET.Element("famliy") # 創建大兒子 # son1 = ET.Element('son', {'name': '兒1'}) son1 = root.makeelement('son', {'name': '兒1'}) # 創建小兒子 # son2 = ET.Element('son', {"name": '兒2'}) son2 = root.makeelement('son', {"name": '兒2'}) # 在大兒子中創建兩個孫子 # grandson1 = ET.Element('grandson', {'name': '兒11'}) grandson1 = son1.makeelement('grandson', {'name': '兒11'}) # grandson2 = ET.Element('grandson', {'name': '兒12'}) grandson2 = son1.makeelement('grandson', {'name': '兒12'}) son1.append(grandson1) son1.append(grandson2) # 把兒子添加到根節點中 root.append(son1) root.append(son1) tree = ET.ElementTree(root) tree.write('oooo.xml',encoding='utf-8', short_empty_elements=False)
創建方式(一)
from xml.etree import ElementTree as ET # 創建根節點 root = ET.Element("famliy") # 創建節點大兒子 son1 = ET.SubElement(root, "son", attrib={'name': '兒1'}) # 創建小兒子 son2 = ET.SubElement(root, "son", attrib={"name": "兒2"}) # 在大兒子中創建一個孫子 grandson1 = ET.SubElement(son1, "age", attrib={'name': '兒11'}) grandson1.text = '孫子' et = ET.ElementTree(root) #生成文檔對象 et.write("test.xml", encoding="utf-8", xml_declaration=True, short_empty_elements=False)
由於原生保存的XML時默認無縮進,如果想要設置縮進的話, 需要修改保存方式:
from xml.etree import ElementTree as ET from xml.dom import minidom def prettify(elem): """將節點轉換成字符串,並添加縮進。 """ rough_string = ET.tostring(elem, 'utf-8') reparsed = minidom.parseString(rough_string) return reparsed.toprettyxml(indent="\t") # 創建根節點 root = ET.Element("famliy") # 創建大兒子 # son1 = ET.Element('son', {'name': '兒1'}) son1 = root.makeelement('son', {'name': '兒1'}) # 創建小兒子 # son2 = ET.Element('son', {"name": '兒2'}) son2 = root.makeelement('son', {"name": '兒2'}) # 在大兒子中創建兩個孫子 # grandson1 = ET.Element('grandson', {'name': '兒11'}) grandson1 = son1.makeelement('grandson', {'name': '兒11'}) # grandson2 = ET.Element('grandson', {'name': '兒12'}) grandson2 = son1.makeelement('grandson', {'name': '兒12'}) son1.append(grandson1) son1.append(grandson2) # 把兒子添加到根節點中 root.append(son1) root.append(son1) raw_str = prettify(root) f = open("xxxoo.xml",'w',encoding='utf-8') f.write(raw_str) f.close()
4、命名空間
詳細介紹,猛擊這里
from xml.etree import ElementTree as ET ET.register_namespace('com',"http://www.company.com") #some name # build a tree structure root = ET.Element("{http://www.company.com}STUFF") body = ET.SubElement(root, "{http://www.company.com}MORE_STUFF", attrib={"{http://www.company.com}hhh": "123"}) body.text = "STUFF EVERYWHERE!" # wrap it in an ElementTree instance, and save as XML tree = ET.ElementTree(root) tree.write("page.xml", xml_declaration=True, encoding='utf-8', method="xml")
PyYAML模塊
Python也可以很容易的處理ymal文檔格式,只不過需要安裝一個模塊,參考文檔:http://pyyaml.org/wiki/PyYAMLDocumentation
configparser模塊
configparser用於處理特定格式的文件如有鍵值對[]等,其本質上是利用open來操作文件。
# 注釋1 ; 注釋2 [section1] # 節點 k1 = v1 # 值 k2:v2 # 值 [section2] # 節點 k1 = v1 # 值
1、獲取所有節點
import configparser config = configparser.ConfigParser() config.read('xxxooo', encoding='utf-8') ret = config.sections() print(ret)
2、獲取指定節點下所有的鍵值對
import configparser config = configparser.ConfigParser() config.read('xxxooo', encoding='utf-8') ret = config.items('section1') print(ret)
3、獲取指定節點下所有的建
import configparser config = configparser.ConfigParser() config.read('xxxooo', encoding='utf-8') ret = config.options('section1') print(ret)
4、獲取指定節點下指定key的值
import configparser config = configparser.ConfigParser() config.read('xxxooo', encoding='utf-8') v = config.get('section1', 'k1') # v = config.getint('section1', 'k1') # v = config.getfloat('section1', 'k1') # v = config.getboolean('section1', 'k1') print(v)
5、檢查、刪除、添加節點
import configparser config = configparser.ConfigParser() config.read('xxxooo', encoding='utf-8') # 檢查 has_sec = config.has_section('section1') print(has_sec) # 添加節點 config.add_section("SEC_1") config.write(open('xxxooo', 'w')) # 刪除節點 config.remove_section("SEC_1") config.write(open('xxxooo', 'w'))
6、檢查、刪除、設置指定組內的鍵值對
import configparser config = configparser.ConfigParser() config.read('xxxooo', encoding='utf-8') # 檢查 has_opt = config.has_option('section1', 'k1') print(has_opt) # 刪除 config.remove_option('section1', 'k1') config.write(open('xxxooo', 'w')) # 設置 config.set('section1', 'k10', "123") config.write(open('xxxooo', 'w'))
shutil模塊
高級的 文件、文件夾、壓縮包 處理模塊,注意當前用戶要是對其他文件或目錄沒有權限會報錯
shutil.copyfileobj(fsrc, fdst[, length]) 將文件內容拷貝到另一個文件中
import shutil shutil.copyfileobj(open('/etc/passwd','r'), open('password', 'w'))
shutil.copyfile(src, dst) 拷貝文件
import shutil shutil.copyfile('/etc/passwd','password1')
shutil.ignore_patterns(*patterns) 忽略某些文件
shutil.copytree(src, dst, symlinks=False, ignore=None) 遞歸的去拷貝文件夾
import shutil shutil.copytree('/etc','etc', ignore=shutil.ignore_patterns('*.conf', 'tmp*'))
shutil.rmtree(path[, ignore_errors[, onerror]]) 遞歸刪除文件夾
import shutil shutil.rmtree('etc')
shutil.move(src, dst) 它類似mv命令,其實就是重命名。
import shutil shutil.move('folder1', 'folder3')
shutil.make_archive(base_name, format,...) 創建壓縮包並返回文件路徑,例如:zip、tar
創建壓縮包並返回文件路徑,例如:zip、tar
- base_name: 壓縮包的文件名,也可以是壓縮包的路徑。只是文件名時,則保存至當前目錄,否則保存至指定路徑,
如:www =>保存至當前路徑
如:/Users/wupeiqi/www =>保存至/Users/wupeiqi/ - format: 壓縮包種類,“zip”, “tar”, “bztar”,“gztar”
- root_dir: 要壓縮的文件夾路徑(默認當前目錄)
- owner: 用戶,默認當前用戶
- group: 組,默認當前組
- logger: 用於記錄日志,通常是logging.Logger對象
import shutil #將 /home/tomcat/ 下的文件打包放置當前程序目錄 ret = shutil.make_archive("test", 'gztar', root_dir='/home/tomcat/') #將 /home/tomcat/ 下的文件打包放置 /home/tomcat/目錄 ret = shutil.make_archive("/home/tomcat/www", 'gztar', root_dir='/home/tomcat/')
shutil 對壓縮包的處理是調用 ZipFile 和 TarFile 兩個模塊來進行的,詳細:
import zipfile # 壓縮 z = zipfile.ZipFile('laxi.zip', 'w') z.write('a.log') z.write('data.data') z.close() # 解壓 z = zipfile.ZipFile('laxi.zip', 'r') z.extractall() z.close()
TarFile
import tarfile # 壓縮 tar = tarfile.open('your.tar','w') tar.add('/Users/wupeiqi/PycharmProjects/bbs2.log', arcname='bbs2.log') tar.add('/Users/wupeiqi/PycharmProjects/cmdb.log', arcname='cmdb.log') tar.close() # 解壓 tar = tarfile.open('your.tar','r') tar.extractall() # 可設置解壓地址 tar.close()