【緣起】
最近兩個月的工作重點是學習CDH（Cloudera's Distribution Including Hadoop）的安裝部署和使用。在閱讀CDH的官方文檔時，其中一句話吸引了我的眼球： Cloudera Manager uses an open source supervisor called supervisord that takes care of redirecting log files, notifying of process failure, setting the effective user ID of the calling process to the right user, and so forth 。CDH自動化部署、中央控制做得如此好，supervisord功不可沒。
在閱讀另一篇教程《Running a Multi-Node Storm Cluster》時，作者也建議在生產環境中使用supervisord進行storm守護進程的管理。
在不同的地方兩次接觸到supervisord這個工具，我對supervisord這個工具充滿了好奇，於是抽出時間自己嘗試了一下。
【產生背景】
在一個分布式環境中，每台機器上可能需要啟動和停止多個進程，使用命令行方式一個一個手動啟動和停止非常麻煩，而且查看每個進程的狀態也很不方便。如果有一個工具能夠實現每台機器上多個進程的簡單高效中心化管理將是非常方便的。於是Supervisord工具應運而生。與Supervisord類似的工具包括monit, daemontools和runit。本文只涉及Supervisord。
【簡介】
supervisord的官網：http://supervisord.org。看懂英文的可以不用看我的博客，直接看文檔就行了，文檔寫得非常好。點個贊！！
Supervisor是一個客戶/服務器系統，它可以在類Unix系統中管理控制大量進程。Supervisor使用python開發，有多年歷史，目前很多生產環境下的服務器都在使用Supervisor。
Supervisor的服務器端稱為supervisord，主要負責在啟動自身時啟動管理的子進程，響應客戶端的命令，重啟崩潰或退出的子進程，記錄子進程stdout和stderr輸出，生成和處理子進程生命周期中的事件。可以在一個配置文件中配置相關參數，包括Supervisord自身的狀態，其管理的各個子進程的相關屬性。配置文件一般位於/etc/supervisord.conf。
Supervisor的客戶端稱為supervisorctl，它提供了一個類shell的接口（即命令行）來使用supervisord服務端提供的功能。通過supervisorctl，用戶可以連接到supervisord服務器進程，獲得服務器進程控制的子進程的狀態，啟動和停止子進程，獲得正在運行的進程列表。客戶端通過Unix域套接字或者TCP套接字與服務端進行通信，服務器端具有身份憑證認證機制，可以有效提升安全性。當客戶端和服務器位於同一台機器上時，客戶端與服務器共用同一個配置文件/etc/supervisord.conf，通過不同標簽來區分兩者的配置。
Supervisor也提供了一個web頁面來查看和管理進程狀態，這個功能用得人比較少。
【平台要求】
Supervisor可以運行在大多數Unix系統上，但不支持在Windows系統上運行。
Supervisor需要Python2.4及以上版本，但任何Python 3版本都不支持。
【安裝】
當前Supervisor的最高版本是3.0，之前嘗試使用2.x版本管理實驗集群中的若干mdrill進程，發現使用客戶端無法有效啟動和停止服務器端管理的各個子進程，從網上搜索錯誤發現2.x版本有一些bug，建議升級到3.0版本。因此我卸載了2.x版本，重新安裝了3.0版本，發現3.0版本很好使。3.0版本相對2.x版本，配置文件不同部分的配置項都發生了變化，詳見官方文檔。
我使用的操作系統是CentOS 6.4， 默認已經安裝了Python，且是2.6.6版本。如果Python沒有安裝，請自行安裝符合要求的版本。
1. 安裝easy_install
sudo yum install python-setuptools-devel
2. 安裝Supervisor
easy_install supervisor
3. 生成配置文件
echo_supervisord_conf > /etc/supervisord.conf
經過上面兩步，Supervisor就安裝好了。配置文件模版已經在/etc/supervisord.conf中生成。當然配置文件的位置不一定放在/etc/supervisord.conf中。在不指定配置文件位置的情況下，supervisord和supervisorctl會按照以下三個順序來搜索配置文件的位置。
- $CWD/supervisord.conf
- $CWD/etc/supervisord.conf
- /etc/supervisord.conf
接下來就可以根據需要修改配置文件中的相應部分，對需要管理的進程進行配置了。具體的配置可以參考下一部分。
【配置】
我在一個三個結點的集群中部署了mdrill，三台主機名稱分別為cma01, cma02, cma03，在cma01和cma02上打算啟動Supervisor進程（這里的Supervisor進程是storm的進程，與supervisor進程管理工具不是一回事），在cma03上打算啟動NimbusServer，Mdrillui和Supervisor三個進程。在三台主機上都安裝supervisor進程控制系統后，分別在每台主機的/etc/supervisord.conf文件中配置每台主機要管理的進程。具體配置如下，其中以；開始的行為注釋行：
[cma01 /etc/supervisord.conf]
; Sample supervisor config file. ; ; For more information on the config file, please see: ; http://supervisord.org/configuration.html ; ; Note: shell expansion ("~" or "$HOME") is not supported. Environment ; variables can be expanded using this syntax: "%(ENV_HOME)s". [unix_http_server] file=/tmp/supervisor.sock ; (the path to the socket file) ;chmod=0700 ; socket file mode (default 0700) ;chown=nobody:nogroup ; socket file uid:gid owner ;username=user ; (default is no username (open server)) ;password=123 ; (default is no password (open server)) [inet_http_server] ; inet (TCP) server disabled by default port=cma01:9001 ; (ip_address:port specifier, *:port for all iface) ;username=user ; (default is no username (open server)) ;password=123 ; (default is no password (open server)) [supervisord] logfile=/tmp/supervisord.log ; (main log file;default $CWD/supervisord.log) logfile_maxbytes=50MB ; (max main logfile bytes b4 rotation;default 50MB) logfile_backups=10 ; (num of main logfile rotation backups;default 10) loglevel=info ; (log level;default info; others: debug,warn,trace) pidfile=/tmp/supervisord.pid ; (supervisord pidfile;default supervisord.pid) nodaemon=false ; (start in foreground if true;default false) minfds=1024 ; (min. avail startup file descriptors;default 1024) minprocs=200 ; (min. avail process descriptors;default 200) ;umask=022 ; (process file creation umask;default 022) ;user=chrism ; (default is current user, required if root) ;identifier=supervisor ; (supervisord identifier, default is 'supervisor') ;directory=/tmp ; (default is not to cd during start) ;nocleanup=true ; (don't clean up tempfiles at start;default false) ;childlogdir=/tmp ; ('AUTO' child log dir, default $TEMP) ;environment=KEY="value" ; (key value pairs to add to environment) ;strip_ansi=false ; (strip ansi escape codes in logs; def. false) ; the below section must remain in the config file for RPC ; (supervisorctl/web interface) to work, additional interfaces may be ; added by defining them in separate rpcinterface: sections [rpcinterface:supervisor] supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface [supervisorctl] serverurl=unix:///tmp/supervisor.sock ; use a unix:// URL for a unix socket ;serverurl=http://127.0.0.1:9001 ; use an http:// url to specify an inet socket ;username=chris ; should be same as http_username if set ;password=123 ; should be same as http_password if set ;prompt=mysupervisor ; cmd line prompt (default "supervisor") ;history_file=~/.sc_history ; use readline history if available ; The below sample program section shows all possible program subsection values, ; create one or more 'real' program: sections to be able to control them under ; supervisor. ;[program:theprogramname] ;command=/bin/cat ; the program (relative uses PATH, can take args) ;process_name=%(program_name)s ; process_name expr (default %(program_name)s) ;numprocs=1 ; number of processes copies to start (def 1) ;directory=/tmp ; directory to cwd to before exec (def no cwd) ;umask=022 ; umask for process (default None) ;priority=999 ; the relative start priority (default 999) ;autostart=true ; start at supervisord start (default: true) ;autorestart=unexpected ; whether/when to restart (default: unexpected) ;startsecs=1 ; number of secs prog must stay running (def. 1) ;startretries=3 ; max # of serial start failures (default 3) ;exitcodes=0,2 ; 'expected' exit codes for process (default 0,2) ;stopsignal=QUIT ; signal used to kill process (default TERM) ;stopwaitsecs=10 ; max num secs to wait b4 SIGKILL (default 10) ;stopasgroup=false ; send stop signal to the UNIX process group (default false) ;killasgroup=false ; SIGKILL the UNIX process group (def false) ;user=chrism ; setuid to this UNIX account to run the program ;redirect_stderr=true ; redirect proc stderr to stdout (default false) ;stdout_logfile=/a/path ; stdout log path, NONE for none; default AUTO ;stdout_logfile_maxbytes=1MB ; max # logfile bytes b4 rotation (default 50MB) ;stdout_logfile_backups=10 ; # of stdout logfile backups (default 10) ;stdout_capture_maxbytes=1MB ; number of bytes in 'capturemode' (default 0) ;stdout_events_enabled=false ; emit events on stdout writes (default false) ;stderr_logfile=/a/path ; stderr log path, NONE for none; default AUTO ;stderr_logfile_maxbytes=1MB ; max # logfile bytes b4 rotation (default 50MB) ;stderr_logfile_backups=10 ; # of stderr logfile backups (default 10) ;stderr_capture_maxbytes=1MB ; number of bytes in 'capturemode' (default 0) ;stderr_events_enabled=false ; emit events on stderr writes (default false) ;environment=A="1",B="2" ; process environment additions (def no adds) ;serverurl=AUTO ; override serverurl computation (childutils) [program:spvs-01] command=/root/alimama/adhoc-core/bin/bluewhale supervisor process_name=%(program_name)s numprocs=1 user=root autostart=true autorestart=true startsecs=5 startretries=3 redirect_stderr=true stdout_logfile=/root/alimama/adhoc-core/bin/supervisor.log stdout_logfile_maxbytes=20MB stdout_logfile_backups=10 stopasgroup=true ; The below sample eventlistener section shows all possible ; eventlistener subsection values, create one or more 'real' ; eventlistener: sections to be able to handle event notifications ; sent by supervisor. ;[eventlistener:theeventlistenername] ;command=/bin/eventlistener ; the program (relative uses PATH, can take args) ;process_name=%(program_name)s ; process_name expr (default %(program_name)s) ;numprocs=1 ; number of processes copies to start (def 1) ;events=EVENT ; event notif. types to subscribe to (req'd) ;buffer_size=10 ; event buffer queue size (default 10) ;directory=/tmp ; directory to cwd to before exec (def no cwd) ;umask=022 ; umask for process (default None) ;priority=-1 ; the relative start priority (default -1) ;autostart=true ; start at supervisord start (default: true) ;autorestart=unexpected ; whether/when to restart (default: unexpected) ;startsecs=1 ; number of secs prog must stay running (def. 1) ;startretries=3 ; max # of serial start failures (default 3) ;exitcodes=0,2 ; 'expected' exit codes for process (default 0,2) ;stopsignal=QUIT ; signal used to kill process (default TERM) ;stopwaitsecs=10 ; max num secs to wait b4 SIGKILL (default 10) ;stopasgroup=false ; send stop signal to the UNIX process group (default false) ;killasgroup=false ; SIGKILL the UNIX process group (def false) ;user=chrism ; setuid to this UNIX account to run the program ;redirect_stderr=true ; redirect proc stderr to stdout (default false) ;stdout_logfile=/a/path ; stdout log path, NONE for none; default AUTO ;stdout_logfile_maxbytes=1MB ; max # logfile bytes b4 rotation (default 50MB) ;stdout_logfile_backups=10 ; # of stdout logfile backups (default 10) ;stdout_events_enabled=false ; emit events on stdout writes (default false) ;stderr_logfile=/a/path ; stderr log path, NONE for none; default AUTO ;stderr_logfile_maxbytes=1MB ; max # logfile bytes b4 rotation (default 50MB) ;stderr_logfile_backups ; # of stderr logfile backups (default 10) ;stderr_events_enabled=false ; emit events on stderr writes (default false) ;environment=A="1",B="2" ; process environment additions ;serverurl=AUTO ; override serverurl computation (childutils) ; The below sample group section shows all possible group values, ; create one or more 'real' group: sections to create "heterogeneous" ; process groups. ;[group:thegroupname] ;programs=progname1,progname2 ; each refers to 'x' in [program:x] definitions ;priority=999 ; the relative start priority (default 999) ; The [include] section can just contain the "files" setting. This ; setting can list multiple files (separated by whitespace or ; newlines). It can also contain wildcards. The filenames are ; interpreted as relative to this file. Included files *cannot* ; include files themselves. ;[include] ;files = relative/directory/*.ini
關鍵部分是program:spvs-01，該部分為主機cma01配置了讓supervisord管理的storm進程Supervisor。
[cma02 /etc/supervisord.conf]
; Sample supervisor config file. ; ; For more information on the config file, please see: ; http://supervisord.org/configuration.html ; ; Note: shell expansion ("~" or "$HOME") is not supported. Environment ; variables can be expanded using this syntax: "%(ENV_HOME)s". [unix_http_server] file=/tmp/supervisor.sock ; (the path to the socket file) ;chmod=0700 ; socket file mode (default 0700) ;chown=nobody:nogroup ; socket file uid:gid owner ;username=user ; (default is no username (open server)) ;password=123 ; (default is no password (open server)) [inet_http_server] ; inet (TCP) server disabled by default port=cma02:9001 ; (ip_address:port specifier, *:port for all iface) ;username=user ; (default is no username (open server)) ;password=123 ; (default is no password (open server)) [supervisord] logfile=/tmp/supervisord.log ; (main log file;default $CWD/supervisord.log) logfile_maxbytes=50MB ; (max main logfile bytes b4 rotation;default 50MB) logfile_backups=10 ; (num of main logfile rotation backups;default 10) loglevel=info ; (log level;default info; others: debug,warn,trace) pidfile=/tmp/supervisord.pid ; (supervisord pidfile;default supervisord.pid) nodaemon=false ; (start in foreground if true;default false) minfds=1024 ; (min. avail startup file descriptors;default 1024) minprocs=200 ; (min. avail process descriptors;default 200) ;umask=022 ; (process file creation umask;default 022) ;user=chrism ; (default is current user, required if root) ;identifier=supervisor ; (supervisord identifier, default is 'supervisor') ;directory=/tmp ; (default is not to cd during start) ;nocleanup=true ; (don't clean up tempfiles at start;default false) ;childlogdir=/tmp ; ('AUTO' child log dir, default $TEMP) ;environment=KEY="value" ; (key value pairs to add to environment) ;strip_ansi=false ; (strip ansi escape codes in logs; def. false) ; the below section must remain in the config file for RPC ; (supervisorctl/web interface) to work, additional interfaces may be ; added by defining them in separate rpcinterface: sections [rpcinterface:supervisor] supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface [supervisorctl] serverurl=unix:///tmp/supervisor.sock ; use a unix:// URL for a unix socket ;serverurl=http://127.0.0.1:9001 ; use an http:// url to specify an inet socket ;username=chris ; should be same as http_username if set ;password=123 ; should be same as http_password if set ;prompt=mysupervisor ; cmd line prompt (default "supervisor") ;history_file=~/.sc_history ; use readline history if available ; The below sample program section shows all possible program subsection values, ; create one or more 'real' program: sections to be able to control them under ; supervisor. ;[program:theprogramname] ;command=/bin/cat ; the program (relative uses PATH, can take args) ;process_name=%(program_name)s ; process_name expr (default %(program_name)s) ;numprocs=1 ; number of processes copies to start (def 1) ;directory=/tmp ; directory to cwd to before exec (def no cwd) ;umask=022 ; umask for process (default None) ;priority=999 ; the relative start priority (default 999) ;autostart=true ; start at supervisord start (default: true) ;autorestart=unexpected ; whether/when to restart (default: unexpected) ;startsecs=1 ; number of secs prog must stay running (def. 1) ;startretries=3 ; max # of serial start failures (default 3) ;exitcodes=0,2 ; 'expected' exit codes for process (default 0,2) ;stopsignal=QUIT ; signal used to kill process (default TERM) ;stopwaitsecs=10 ; max num secs to wait b4 SIGKILL (default 10) ;stopasgroup=false ; send stop signal to the UNIX process group (default false) ;killasgroup=false ; SIGKILL the UNIX process group (def false) ;user=chrism ; setuid to this UNIX account to run the program ;redirect_stderr=true ; redirect proc stderr to stdout (default false) ;stdout_logfile=/a/path ; stdout log path, NONE for none; default AUTO ;stdout_logfile_maxbytes=1MB ; max # logfile bytes b4 rotation (default 50MB) ;stdout_logfile_backups=10 ; # of stdout logfile backups (default 10) ;stdout_capture_maxbytes=1MB ; number of bytes in 'capturemode' (default 0) ;stdout_events_enabled=false ; emit events on stdout writes (default false) ;stderr_logfile=/a/path ; stderr log path, NONE for none; default AUTO ;stderr_logfile_maxbytes=1MB ; max # logfile bytes b4 rotation (default 50MB) ;stderr_logfile_backups=10 ; # of stderr logfile backups (default 10) ;stderr_capture_maxbytes=1MB ; number of bytes in 'capturemode' (default 0) ;stderr_events_enabled=false ; emit events on stderr writes (default false) ;environment=A="1",B="2" ; process environment additions (def no adds) ;serverurl=AUTO ; override serverurl computation (childutils) [program:spvs-02] command=/root/alimama/adhoc-core/bin/bluewhale supervisor process_name=%(program_name)s numprocs=1 user=root autostart=true autorestart=true startsecs=5 startretries=3 redirect_stderr=true stdout_logfile=/root/alimama/adhoc-core/bin/supervisor.log stdout_logfile_maxbytes=20MB stdout_logfile_backups=10 stopasgroup=true ; The below sample eventlistener section shows all possible ; eventlistener subsection values, create one or more 'real' ; eventlistener: sections to be able to handle event notifications ; sent by supervisor. ;[eventlistener:theeventlistenername] ;command=/bin/eventlistener ; the program (relative uses PATH, can take args) ;process_name=%(program_name)s ; process_name expr (default %(program_name)s) ;numprocs=1 ; number of processes copies to start (def 1) ;events=EVENT ; event notif. types to subscribe to (req'd) ;buffer_size=10 ; event buffer queue size (default 10) ;directory=/tmp ; directory to cwd to before exec (def no cwd) ;umask=022 ; umask for process (default None) ;priority=-1 ; the relative start priority (default -1) ;autostart=true ; start at supervisord start (default: true) ;autorestart=unexpected ; whether/when to restart (default: unexpected) ;startsecs=1 ; number of secs prog must stay running (def. 1) ;startretries=3 ; max # of serial start failures (default 3) ;exitcodes=0,2 ; 'expected' exit codes for process (default 0,2) ;stopsignal=QUIT ; signal used to kill process (default TERM) ;stopwaitsecs=10 ; max num secs to wait b4 SIGKILL (default 10) ;stopasgroup=false ; send stop signal to the UNIX process group (default false) ;killasgroup=false ; SIGKILL the UNIX process group (def false) ;user=chrism ; setuid to this UNIX account to run the program ;redirect_stderr=true ; redirect proc stderr to stdout (default false) ;stdout_logfile=/a/path ; stdout log path, NONE for none; default AUTO ;stdout_logfile_maxbytes=1MB ; max # logfile bytes b4 rotation (default 50MB) ;stdout_logfile_backups=10 ; # of stdout logfile backups (default 10) ;stdout_events_enabled=false ; emit events on stdout writes (default false) ;stderr_logfile=/a/path ; stderr log path, NONE for none; default AUTO ;stderr_logfile_maxbytes=1MB ; max # logfile bytes b4 rotation (default 50MB) ;stderr_logfile_backups ; # of stderr logfile backups (default 10) ;stderr_events_enabled=false ; emit events on stderr writes (default false) ;environment=A="1",B="2" ; process environment additions ;serverurl=AUTO ; override serverurl computation (childutils) ; The below sample group section shows all possible group values, ; create one or more 'real' group: sections to create "heterogeneous" ; process groups. ;[group:thegroupname] ;programs=progname1,progname2 ; each refers to 'x' in [program:x] definitions ;priority=999 ; the relative start priority (default 999) ; The [include] section can just contain the "files" setting. This ; setting can list multiple files (separated by whitespace or ; newlines). It can also contain wildcards. The filenames are ; interpreted as relative to this file. Included files *cannot* ; include files themselves. ;[include] ;files = relative/directory/*.ini
關鍵部分是program:spvs-02，該部分為主機cma02配置了讓supervisord管理的storm進程Supervisor。
[cma03 /etc/supervisord.conf]
; Sample supervisor config file. ; ; For more information on the config file, please see: ; http://supervisord.org/configuration.html ; ; Note: shell expansion ("~" or "$HOME") is not supported. Environment ; variables can be expanded using this syntax: "%(ENV_HOME)s". [unix_http_server] file=/tmp/supervisor.sock ; (the path to the socket file) ;chmod=0700 ; socket file mode (default 0700) ;chown=nobody:nogroup ; socket file uid:gid owner ;username=user ; (default is no username (open server)) ;password=123 ; (default is no password (open server)) [inet_http_server] ; inet (TCP) server disabled by default port=cma03:9001 ; (ip_address:port specifier, *:port for all iface) ;username=user ; (default is no username (open server)) ;password=123 ; (default is no password (open server)) [supervisord] logfile=/tmp/supervisord.log ; (main log file;default $CWD/supervisord.log) logfile_maxbytes=50MB ; (max main logfile bytes b4 rotation;default 50MB) logfile_backups=10 ; (num of main logfile rotation backups;default 10) loglevel=info ; (log level;default info; others: debug,warn,trace) pidfile=/tmp/supervisord.pid ; (supervisord pidfile;default supervisord.pid) nodaemon=false ; (start in foreground if true;default false) minfds=1024 ; (min. avail startup file descriptors;default 1024) minprocs=200 ; (min. avail process descriptors;default 200) ;umask=022 ; (process file creation umask;default 022) ;user=chrism ; (default is current user, required if root) ;identifier=supervisor ; (supervisord identifier, default is 'supervisor') ;directory=/tmp ; (default is not to cd during start) ;nocleanup=true ; (don't clean up tempfiles at start;default false) ;childlogdir=/tmp ; ('AUTO' child log dir, default $TEMP) ;environment=KEY="value" ; (key value pairs to add to environment) ;strip_ansi=false ; (strip ansi escape codes in logs; def. false) ; the below section must remain in the config file for RPC ; (supervisorctl/web interface) to work, additional interfaces may be ; added by defining them in separate rpcinterface: sections [rpcinterface:supervisor] supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface [supervisorctl] serverurl=unix:///tmp/supervisor.sock ; use a unix:// URL for a unix socket ;serverurl=http://127.0.0.1:9001 ; use an http:// url to specify an inet socket ;username=chris ; should be same as http_username if set ;password=123 ; should be same as http_password if set ;prompt=mysupervisor ; cmd line prompt (default "supervisor") ;history_file=~/.sc_history ; use readline history if available ; The below sample program section shows all possible program subsection values, ; create one or more 'real' program: sections to be able to control them under ; supervisor. ;[program:theprogramname] ;command=/bin/cat ; the program (relative uses PATH, can take args) ;process_name=%(program_name)s ; process_name expr (default %(program_name)s) ;numprocs=1 ; number of processes copies to start (def 1) ;directory=/tmp ; directory to cwd to before exec (def no cwd) ;umask=022 ; umask for process (default None) ;priority=999 ; the relative start priority (default 999) ;autostart=true ; start at supervisord start (default: true) ;autorestart=unexpected ; whether/when to restart (default: unexpected) ;startsecs=1 ; number of secs prog must stay running (def. 1) ;startretries=3 ; max # of serial start failures (default 3) ;exitcodes=0,2 ; 'expected' exit codes for process (default 0,2) ;stopsignal=QUIT ; signal used to kill process (default TERM) ;stopwaitsecs=10 ; max num secs to wait b4 SIGKILL (default 10) ;stopasgroup=false ; send stop signal to the UNIX process group (default false) ;killasgroup=false ; SIGKILL the UNIX process group (def false) ;user=chrism ; setuid to this UNIX account to run the program ;redirect_stderr=true ; redirect proc stderr to stdout (default false) ;stdout_logfile=/a/path ; stdout log path, NONE for none; default AUTO ;stdout_logfile_maxbytes=1MB ; max # logfile bytes b4 rotation (default 50MB) ;stdout_logfile_backups=10 ; # of stdout logfile backups (default 10) ;stdout_capture_maxbytes=1MB ; number of bytes in 'capturemode' (default 0) ;stdout_events_enabled=false ; emit events on stdout writes (default false) ;stderr_logfile=/a/path ; stderr log path, NONE for none; default AUTO ;stderr_logfile_maxbytes=1MB ; max # logfile bytes b4 rotation (default 50MB) ;stderr_logfile_backups=10 ; # of stderr logfile backups (default 10) ;stderr_capture_maxbytes=1MB ; number of bytes in 'capturemode' (default 0) ;stderr_events_enabled=false ; emit events on stderr writes (default false) ;environment=A="1",B="2" ; process environment additions (def no adds) ;serverurl=AUTO ; override serverurl computation (childutils) [program:nmbs] command=/root/alimama/adhoc-core/bin/bluewhale nimbus process_name=%(program_name)s numprocs=1 user=root priority=700 autostart=true autorestart=true startsecs=5 startretries=3 redirect_stderr=true stdout_logfile=/root/alimama/adhoc-core/bin/nimbus.log stdout_logfile_maxbytes=20MB stdout_logfile_backups=10 stopasgroup=true [program:spvs-03] command=/root/alimama/adhoc-core/bin/bluewhale supervisor process_name=%(program_name)s numprocs=1 user=root priority=800 autostart=true autorestart=true startsecs=5 startretries=3 redirect_stderr=true stdout_logfile=/root/alimama/adhoc-core/bin/supervisor.log stdout_logfile_maxbytes=20MB stdout_logfile_backups=10 stopasgroup=true [program:ui] command=/root/alimama/adhoc-core/bin/bluewhale mdrillui 1107 /root/alimama/adhoc-core/lib/adhoc-web-0.18-beta.jar process_name=%(program_name)s numprocs=1 user=root priority=900 autostart=true autorestart=true startsecs=5 startretries=3 redirect_stderr=true stdout_logfile=/root/alimama/adhoc-core/bin/ui.log stdout_logfile_maxbytes=20MB stdout_logfile_backups=10 stopasgroup=true ; The below sample eventlistener section shows all possible ; eventlistener subsection values, create one or more 'real' ; eventlistener: sections to be able to handle event notifications ; sent by supervisor. ;[eventlistener:theeventlistenername] ;command=/bin/eventlistener ; the program (relative uses PATH, can take args) ;process_name=%(program_name)s ; process_name expr (default %(program_name)s) ;numprocs=1 ; number of processes copies to start (def 1) ;events=EVENT ; event notif. types to subscribe to (req'd) ;buffer_size=10 ; event buffer queue size (default 10) ;directory=/tmp ; directory to cwd to before exec (def no cwd) ;umask=022 ; umask for process (default None) ;priority=-1 ; the relative start priority (default -1) ;autostart=true ; start at supervisord start (default: true) ;autorestart=unexpected ; whether/when to restart (default: unexpected) ;startsecs=1 ; number of secs prog must stay running (def. 1) ;startretries=3 ; max # of serial start failures (default 3) ;exitcodes=0,2 ; 'expected' exit codes for process (default 0,2) ;stopsignal=QUIT ; signal used to kill process (default TERM) ;stopwaitsecs=10 ; max num secs to wait b4 SIGKILL (default 10) ;stopasgroup=false ; send stop signal to the UNIX process group (default false) ;killasgroup=false ; SIGKILL the UNIX process group (def false) ;user=chrism ; setuid to this UNIX account to run the program ;redirect_stderr=true ; redirect proc stderr to stdout (default false) ;stdout_logfile=/a/path ; stdout log path, NONE for none; default AUTO ;stdout_logfile_maxbytes=1MB ; max # logfile bytes b4 rotation (default 50MB) ;stdout_logfile_backups=10 ; # of stdout logfile backups (default 10) ;stdout_events_enabled=false ; emit events on stdout writes (default false) ;stderr_logfile=/a/path ; stderr log path, NONE for none; default AUTO ;stderr_logfile_maxbytes=1MB ; max # logfile bytes b4 rotation (default 50MB) ;stderr_logfile_backups ; # of stderr logfile backups (default 10) ;stderr_events_enabled=false ; emit events on stderr writes (default false) ;environment=A="1",B="2" ; process environment additions ;serverurl=AUTO ; override serverurl computation (childutils) ; The below sample group section shows all possible group values, ; create one or more 'real' group: sections to create "heterogeneous" ; process groups. ;[group:thegroupname] ;programs=progname1,progname2 ; each refers to 'x' in [program:x] definitions ;priority=999 ; the relative start priority (default 999) ; The [include] section can just contain the "files" setting. This ; setting can list multiple files (separated by whitespace or ; newlines). It can also contain wildcards. The filenames are ; interpreted as relative to this file. Included files *cannot* ; include files themselves. ;[include] ;files = relative/directory/*.ini
關鍵部分是program:nmbs, program:spvs-03和program:ui，這三部分為主機cma03配置了讓supervisord管理的storm進程NimbusServer, Supervisor和Mdrillui。
有幾個配置項值得解釋一下，可以根據需要自行設置。
1. stopasgroup=true。這一配置項的作用是：如果supervisord管理的進程px又產生了若干子進程，使用supervisorctl停止px進程，停止信號會傳播給px產生的所有子進程，確保子進程也一起停止。這一配置項對希望停止所有進程的需求是非常有用的。
2. autostart=true。這一配置項的作用是：當啟動supervisord的時候會將該配置項設置為true的所有進程自動啟動。
【啟動supervisord】
確保配置無誤后可以在每台主機上使用下面的命令啟動supervisor的服務器端supervisord
supervisord
【停止supervisord】
supervisorctl shutdown
【重新加載配置文件】
supervisorctl reload
【進程管理】
1. 啟動supervisord管理的所有進程
supervisorctl start all
2. 停止supervisord管理的所有進程
supervisorctl stop all
3. 啟動supervisord管理的某一個特定進程
supervisorctl start program-name // program-name為[program:xx]中的xx
4. 停止supervisord管理的某一個特定進程
supervisorctl stop program-name // program-name為[program:xx]中的xx
5. 重啟所有進程或所有進程
supervisorctl restart all // 重啟所有
supervisorctl reatart program-name // 重啟某一進程，program-name為[program:xx]中的xx
6. 查看supervisord當前管理的所有進程的狀態
supervisorctl status
【遇到問題及解決方案】
在使用命令supervisorctl start all啟動控制進程時，遇到如下錯誤
unix:///tmp/supervisor.sock no such file
出現上述錯誤的原因是supervisord並未啟動，只要在命令行中使用命令sudo supervisord啟動supervisord即可。
【遺留問題】
當集群規模擴大后，登錄到每台主機使用supervior控制進程也是很麻煩的，能不能用一台主機作為客戶端，同時連接集群中的所有主機，以一種中心化的方式統一管理集群中的所有進程？之前一直使用的方式是使用交互式工具expect。supervisor本身有沒有提供一種機制實現集群中所有進程的中央化管理？
目前測試成功的是使用一台主機作為客戶端（supervisorctl），控制另一台服務器(supervisord)主機上的狀態。方法是在服務器端配置[inet_http_server]部分，開啟TCP端口監聽。客戶端配置[supervisorctl]部分，指定服務器端的serverurl，連接服務器端監聽的端口。但是一個客戶端只能連接一個服務器，無法指定多個服務器。