Puppet是當前devops中常用於管理系統配置和應用部署，多數會使用其C/S架構的方式來進行部署，其中puppetmaster是集群中配置管理的核心節點。在實際的生產環境中，如果因為master節點性能不夠或者發生意外宕機，可能會影響到實際業務，因此維護一個高可用和可擴展的puppetmaster池子是一個首要任務。
這里我使用了一種常規的方案：前端使用apache/nginx做負載均衡，使用packmaker/keepalived來做健康檢查和故障切換，來做HA，后端起多個puppetmaster實例做橫向擴展，來提高處理能力。
方案驗證
這里，我將在在每台Master Node上起兩個puppetmaster實例，前端使用Apache作負載均衡，keepalived做健康檢查。唯一的難點是證書同步問題，在部署中將把我們將證書設成自動認證，只接受fqdn是*.clustername.ustack.com的機器，就不需同步證書了。
| IP | 主機名 | 角色 | vip |
|---|---|---|---|
| 192.168.1.53 | ha-puppet1.ustack.com | puppet master | 192.168.1.103 |
| 192.168.1.54 | ha-puppet2.ustack.com | puppet master | 192.168.1.104 |
配置細節
客戶端 配置文件
[main] logdir=/var/log/puppet vardir=/var/lib/puppet ssldir=/var/lib/puppet/ssl rundir=/var/run/puppet factpath=$vardir/lib/facter templatedir=$confdir/templates pluginsync=true [agent] server = ha-puppet.ustack.com report = true pluginsync = true listen = true runinterval = 300
Master node的配置選項
[master]
autosign = $confdir/autosign.conf { mode = 664 }
autosign.conf
*.ustack.com
服務器端配置信息
192.168.2.53
loadbalancer配置
apache的proxy監聽在8140端口，后面可以配置多個puppetmaster進程
<Proxy balancer://puppetmaster>
BalancerMember http://127.0.0.1:18140
BalancerMember http://127.0.0.1:18141
</Proxy>
Listen 8140
<VirtualHost *:8140>
SSLEngine on
SSLProtocol -ALL +SSLv3 +TLSv1
SSLCipherSuite ALL:!ADH:RC4+RSA:+HIGH:+MEDIUM:-LOW:-SSLv2:-EXP
SSLCertificateFile /var/lib/puppet/ssl/certs/ha-puppet.ustack.com.pem
SSLCertificateKeyFile /var/lib/puppet/ssl/private_keys/ha-puppet.ustack.com.pem
SSLCertificateChainFile /var/lib/puppet/ssl/ca/ca_crt.pem
SSLCACertificateFile /var/lib/puppet/ssl/ca/ca_crt.pem
# CRL checking should be enabled; if you have problems with Apache complaining about the CRL, disable the next line
SSLCARevocationFile /var/lib/puppet/ssl/ca/ca_crl.pem
SSLVerifyClient optional
SSLVerifyDepth 1
SSLOptions +StdEnvVars +ExportCertData
# The following client headers allow the same configuration to work with Pound.
RequestHeader set X-SSL-Subject %{SSL_CLIENT_S_DN}e
RequestHeader set X-Client-DN %{SSL_CLIENT_S_DN}e
RequestHeader set X-Client-Verify %{SSL_CLIENT_VERIFY}e
RequestHeader unset X-Forwarded-For
# you probably want to tune these settings
PassengerHighPerformance on
PassengerMaxPoolSize 12
PassengerPoolIdleTime 1500
# PassengerMaxRequests 1000
PassengerStatThrottleRate 120
RackAutoDetect On
DocumentRoot /etc/puppet/rack/public/
<Directory /etc/puppet/rack>
Options None
AllowOverride None
Order allow,deny
allow from all
</Directory>
<Location />
SetHandler balancer-manager
Order allow,deny
Allow from all
</Location>
ProxyPass / balancer://puppetmaster/
ProxyPassReverse / balancer://puppetmaster/
ProxyPreserveHost On
CustomLog /var/log/httpd/balance-8140-access.log combined
ErrorLog /var/log/httpd/balance-8140-error.log
CustomLog /var/log/httpd/balancer_ssl_requests.log "%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x \"%r\" %b"
</VirtualHost>
第一個puppetmaster的vhost配置文件，puppetmaster實例的數量可以水平擴展：
Listen 18140
<VirtualHost *:18140>
SSLEngine off
# The following client headers allow the same configuration to work with Pound.
SetEnvIf set X-Client-Verify "(.*)" SSL_CLIENT_VERIFY=$1
SetEnvIf set X-Client-DN "(.*)" SSL_CLIENT_S_DN=$1
# you probably want to tune these settings
PassengerHighPerformance on
PassengerMaxPoolSize 12
PassengerPoolIdleTime 1500
# PassengerMaxRequests 1000
PassengerStatThrottleRate 120
RackAutoDetect On
DocumentRoot /etc/puppet/rack/18140/public/
<Directory /etc/puppet/rack/18140/>
Options None
AllowOverride None
Order allow,deny
allow from all
</Directory>
CustomLog /var/log/httpd/puppetmaster-18140-access.log combined
ErrorLog /var/log/httpd/puppetmaster-18140-error.log
</VirtualHost>
rack配置文件
拷貝rack配置文件給第一個puppetmaster：
rsync -avxH /etc/puppet/rack/{,18140}/
Keepalived配置文件
! Configuration File for keepalived
global_defs {
notification_email {
acassen@firewall.loc
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server localhost
smtp_connect_timeout 30
router_id route-45
}
vrrp_script chk_http_port {
script "/usr/bin/killall -0 httpd"
interval 2
weight 2
}
vrrp_instance 45 {
virtual_router_id 45
priority 100
state BACKUP
interface eth0
virtual_ipaddress {
192.168.2.103
}
track_script {
chk_http_port
}
}
vrrp_instance 46 {
virtual_router_id 46
priority 101
state MASTER
interface eth0
virtual_ipaddress {
192.168.2.104
}
track_script {
chk_http_port
}
}
192.168.2.54
apache的配置文件和53相同，唯一區別就是keepalived的配置文件上的IP地址互為熱備，在此就不再贅述。
驗證
在dnspod上綁定192.168.1.103/4到ha-puppet.ustack.com。
並在ha-puppet1上使用以下manifests文件：
node /default/ {
notify {'Hello,I am Master 1':}
}
node 'nginx.novalocal' inherits default{}
在ha-puppet2上使用以下manifests文件：
node /default/ {
notify {'Hello,I am Master 2':}
}
node 'nginx.novalocal' inherits default{}
測試結果，前兩次是：
notice: Hello,I am Master 2 notice: /Stage[main]//Node[default]/Notify[Hello,I am Master 2]/message: defined 'message' as 'Hello,I am Master 2' notice: Finished catalog run in 0.12 seconds
隨后出現：
notice: Hello,I am Master 1 notice: /Stage[main]//Node[default]/Notify[Hello,I am Master 1]/message: defined 'message' as 'Hello,I am Master 1' notice: Finished catalog run in 0.10 seconds
結論
以上方案驗證通過，手動部署很簡單，難點在於把它們設計成puppet module來進行部署時，需要考慮到所有的服務器需要使用同一個證書，因此需要在啟動第一台puppetmaster的時候，根據設定的fqdn生成證書，然后修改hostname，同時把證書和配置文件同步到其他服務器上去。