在python中使用zookeeper管理你的應用集群 | -豬之哀傷的Blog-

在python中使用zookeeper管理你的應用集群

Type: 技術相關 - Posted at: 2011/09/09 12:43

Tags :
python
zookeeper

簡介：

Zookeeper 分布式服務框架是 Apache Hadoop 的一個子項目，它主要是用來解決分布式應用中經常遇到的一些數據管理問題，如：統一命名服務、狀態同步服務、集群管理、分布式應用配置項的管理等。

具體簡介可以參照這篇文章。

zkpython的安裝：

python中有一個zkpython的包，是基於zookeeper的c-client開發的，所以安裝的時候需要先安裝zookeeper的c客戶端。安裝步驟如下：

?

1 2 3 4 5 6 7 8 9 10 11 12 13

# 首先下載zookeeper wget http: //labs .renren.com /apache-mirror//zookeeper/zookeeper-3 .3.3 /zookeeper-3 .3.3. tar .gz tar xzvf zookeeper-3.3.3. tar .gz cd zookeeper-3.3.3 /src/c/ . /configure make make install   # 然后下載zkpython wget http: //pypi .python.org /packages/source/z/zkpython/zkpython-0 .4. tar .gz #md5=3de220615aaddf57f1462b78d32477f9 tar xzvf zkpython-0.4. tar .gz cd zkpython-0.4 python setup.py install

這樣就完成了zkpython的安裝。

一個簡單的demo：

之后讓我們來寫一個簡單的demo吧。（demo中用到的zkclient.py: https://github.com/piglei/zkpython_example/blob/master/zkclient.py）

?

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99

# coding: utf-8 import logging from os.path import basename, join   from zkclient import ZKClient, zookeeper, watchmethod   logging.basicConfig(      level = logging.DEBUG,      format = "[%(asctime)s] %(levelname)-8s %(message)s" )   log = logging   class GJZookeeper( object ):        ZK_HOST = "localhost:2181"      ROOT = "/app"      WORKERS_PATH = join(ROOT, "workers" )      MASTERS_NUM = 1      TIMEOUT = 10000        def __init__( self , verbose = True ):          self .VERBOSE = verbose          self .masters = []          self .is_master = False          self .path = None            self .zk = ZKClient( self .ZK_HOST, timeout = self .TIMEOUT)          self .say( "login ok!" )          # init          self .__init_zk()          # register          self .register()        def __init_zk( self ):          """          create the zookeeper node if not exist          """          nodes = ( self .ROOT, self .WORKERS_PATH)          for node in nodes:              if not self .zk.exists(node):                  try :                      self .zk.create(node, "")                  except :                      pass        @property      def is_slave( self ):          return not self .is_master        def register( self ):          """          register a node for this worker          """          self .path = self .zk.create( self .WORKERS_PATH + "/worker" , "1" , flags = zookeeper.EPHEMERAL | zookeeper.SEQUENCE)          self .path = basename( self .path)          self .say( "register ok! I'm %s" % self .path)          # check who is the master          self .get_master()        def get_master( self ):          """          get children, and check who is the smallest child          """          @watchmethod          def watcher(event):              self .say( "child changed, try to get master again." )              self .get_master()            children = self .zk.get_children( self .WORKERS_PATH, watcher)          children.sort()          self .say( "%s's children: %s" % ( self .WORKERS_PATH, children))            # check if I'm master          self .masters = children[: self .MASTERS_NUM]          if self .path in self .masters:              self .is_master = True              self .say( "I've become master!" )          else :              self .say( "%s is masters, I'm slave" % self .masters)          def say( self , msg):          """          print messages to screen          """          if self .VERBOSE:              if self .path:                  log.info( "[ %s(%s) ] %s" % ( self .path, "master" if self .is_master else "slave" , msg))              else :                  log.info(msg)   def main():      gj_zookeeper = GJZookeeper()   if __name__ = = "__main__" :      main()      import time      time.sleep( 1000 )

這個簡單的demo所做的事情，就是通過在zookeeper的/app/workers節點下建立臨時的子節點( flags=zookeeper.EPHEMERAL | zookeeper.SEQUENCE )，每次create完成之后檢查自己是不是在最小的MASTERS_NUM(例子中為1，即單master)里。如果是的話，作為master運行，否則的話，作為slave運行。

這樣的話，當我們的master掛掉以后，與zookeeper之間的連接也會中斷，過了指定的TIMEOUT以后，master之前在worker下的子節點就會被刪除，於是slave節點之前設置的watcher會被觸發，再次檢查自己是否為master，如果是的話則完成切換。

demo運行結果：

?

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

# 第一個實例 Connected in 20 ms, handle is 0 [2011-09-09 12:40:43,702] INFO     login ok! Node /app/workers/worker created in 4 ms [2011-09-09 12:40:43,708] INFO     [ worker0000000022(slave) ] register ok! I'm worker0000000022 [2011-09-09 12:40:43,709] INFO     [ worker0000000022(slave) ] /app/workers's children: ['worker0000000022'] [2011-09-09 12:40:43,709] INFO     [ worker0000000022(master) ] I've become master!   # 這時再起第二個實例 Connected in 64 ms, handle is 0 [2011-09-09 12:43:08,334] INFO     login ok! Node /app/workers/worker created in 11 ms [2011-09-09 12:43:08,346] INFO     [ worker0000000023(slave) ] register ok! I'm worker0000000023 [2011-09-09 12:43:08,347] INFO     [ worker0000000023(slave) ] /app/workers's children: ['worker0000000022', 'worker0000000023'] [2011-09-09 12:43:08,347] INFO     [ worker0000000023(slave) ] ['worker0000000022'] is masters, I'm slave   # 殺掉master，第二個實例發生的變化 [2011-09-09 12:44:06,016] INFO     [ worker0000000023(slave) ] child changed, try to get master again. [2011-09-09 12:44:06,017] INFO     [ worker0000000023(slave) ] /app/workers's children: ['worker0000000023'] [2011-09-09 12:44:06,017] INFO     [ worker0000000023(master) ] I've become master!

 

demo代碼地址： https://github.com/piglei/zkpython_example